2024-09-18T11:05:08.668 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-18T11:05:08.681 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T11:05:08.798 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910935 branch: wip-bharath11-testing-2024-09-18-1115-quincy description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} email: skanta@redhat.com first_in_suite: false job_id: '7910935' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath11-testing-2024-09-18-1115-quincy ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 workunit: branch: wip-bharath11-testing-2024-09-18-1115-quincy sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 4631 sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath11-testing-2024-09-18-1115-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 targets: smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLS3EtqxiHHroj6t4xCSMfkiyJpK2mzUP/qA5oxwdMsMftC4mstGe+p5YQOWOgUgZ+2PqJ59PuD1gj3BXOhz3O4= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-18_09:13:00 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-18T11:05:08.798 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa; will attempt to use it 2024-09-18T11:05:08.799 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa/tasks 2024-09-18T11:05:08.799 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-18T11:05:08.801 INFO:teuthology.task.internal:Checking packages... 2024-09-18T11:05:08.823 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '7061ea6ca50908d104e9f3979dc615e118d9e153' 2024-09-18T11:05:08.823 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-18T11:05:08.823 INFO:teuthology.packaging:ref: None 2024-09-18T11:05:08.823 INFO:teuthology.packaging:tag: None 2024-09-18T11:05:08.823 INFO:teuthology.packaging:branch: wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T11:05:08.823 INFO:teuthology.packaging:sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:05:08.823 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T11:05:08.961 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1645.g7061ea6c 2024-09-18T11:05:08.963 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-18T11:05:08.969 INFO:teuthology.task.internal:no buildpackages task found 2024-09-18T11:05:08.969 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-18T11:05:09.112 INFO:teuthology.task.internal:Saving configuration 2024-09-18T11:05:09.121 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-18T11:05:09.198 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-18T11:05:09.221 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910935', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 10:59:32.243122', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLS3EtqxiHHroj6t4xCSMfkiyJpK2mzUP/qA5oxwdMsMftC4mstGe+p5YQOWOgUgZ+2PqJ59PuD1gj3BXOhz3O4='} 2024-09-18T11:05:09.222 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-18T11:05:09.292 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-18T11:05:09.292 INFO:teuthology.run_tasks:Running task console_log... 2024-09-18T11:05:09.388 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f57bb027ac0>, signals=[15]) 2024-09-18T11:05:09.388 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-18T11:05:09.465 INFO:teuthology.task.internal:Opening connections... 2024-09-18T11:05:09.465 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2024-09-18T11:05:09.466 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:05:09.544 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-18T11:05:09.563 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2024-09-18T11:05:09.626 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2024-09-18T11:05:09.626 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2024-09-18T11:05:09.683 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="CentOS Stream" 2024-09-18T11:05:09.684 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="9" 2024-09-18T11:05:09.684 INFO:teuthology.orchestra.run.smithi157.stdout:ID="centos" 2024-09-18T11:05:09.684 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="rhel fedora" 2024-09-18T11:05:09.684 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="9" 2024-09-18T11:05:09.684 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el9" 2024-09-18T11:05:09.684 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-18T11:05:09.684 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31" 2024-09-18T11:05:09.684 INFO:teuthology.orchestra.run.smithi157.stdout:LOGO="fedora-logo-icon" 2024-09-18T11:05:09.684 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-18T11:05:09.684 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://centos.org/" 2024-09-18T11:05:09.685 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-18T11:05:09.685 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-18T11:05:09.685 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-18T11:05:09.685 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2024-09-18T11:05:09.710 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-18T11:05:09.718 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-18T11:05:09.824 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-18T11:05:09.825 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2024-09-18T11:05:09.841 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-18T11:05:09.929 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-18T11:05:09.929 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2024-09-18T11:05:09.961 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-18T11:05:10.028 INFO:teuthology.run_tasks:Running task kernel... 2024-09-18T11:05:10.071 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-18T11:05:10.071 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-18T11:05:10.071 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T11:05:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:05:10.088 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-09-18T11:05:10.145 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-503.el9.x86_64 2024-09-18T11:05:10.146 INFO:teuthology.task.kernel:Running kernel on smithi157: 5.14.0-503.el9.x86_64 2024-09-18T11:05:10.146 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum install -y kernel 2024-09-18T11:05:13.966 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.3 MB 00:01 2024-09-18T11:05:32.790 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 20 MB 00:17 2024-09-18T11:05:38.165 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.5 MB 00:01 2024-09-18T11:05:39.833 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 19 kB 00:00 2024-09-18T11:05:40.870 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-18T11:05:46.218 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-18T11:05:48.066 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-18T11:05:48.066 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-18T11:05:48.148 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:05:48.159 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:05:48.159 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-09-18T11:05:48.159 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:05:48.160 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:05:48.160 INFO:teuthology.orchestra.run.smithi157.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-18T11:05:48.160 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-18T11:05:48.160 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-18T11:05:48.160 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-18T11:05:48.160 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-18T11:05:48.160 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:05:48.160 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:05:48.160 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:05:48.160 INFO:teuthology.orchestra.run.smithi157.stdout:Install 4 Packages 2024-09-18T11:05:48.160 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:05:48.161 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 79 M 2024-09-18T11:05:48.161 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 126 M 2024-09-18T11:05:48.161 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:05:48.577 INFO:teuthology.orchestra.run.smithi157.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 122 kB/s | 36 kB 00:00 2024-09-18T11:06:05.084 INFO:teuthology.orchestra.run.smithi157.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 954 kB/s | 16 MB 00:16 2024-09-18T11:06:21.432 INFO:teuthology.orchestra.run.smithi157.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 891 kB/s | 29 MB 00:32 2024-09-18T11:06:26.368 INFO:teuthology.orchestra.run.smithi157.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 929 kB/s | 35 MB 00:38 2024-09-18T11:06:26.368 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:06:26.369 INFO:teuthology.orchestra.run.smithi157.stdout:Total 2.1 MB/s | 79 MB 00:38 2024-09-18T11:06:27.038 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:06:27.411 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:06:27.411 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:06:28.351 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:06:28.351 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:06:30.777 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:06:31.433 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-18T11:06:31.524 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T11:06:34.279 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T11:06:34.411 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T11:06:39.418 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T11:06:39.561 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T11:06:44.441 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T11:07:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T11:07:22.627 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-18T11:07:22.956 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T11:07:22.957 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-18T11:07:22.957 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T11:07:22.957 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T11:07:23.790 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T11:07:23.790 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:07:23.790 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:07:23.790 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-18T11:07:23.790 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-18T11:07:23.790 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-18T11:07:23.790 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-18T11:07:23.790 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:07:23.790 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:07:24.151 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T11:07:24.244 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T11:07:24.244 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T11:07:24.245 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-18T11:07:24.245 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T11:07:24.245 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T11:07:24.245 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-18T11:07:24.245 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T11:07:24.245 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T11:07:24.303 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T11:07:24.304 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T11:07:24.304 DEBUG:teuthology.orchestra.run.smithi157:> sudo rpm -qi grub2-tools 2024-09-18T11:07:24.344 INFO:teuthology.orchestra.run.smithi157.stdout:Name : grub2-tools 2024-09-18T11:07:24.344 INFO:teuthology.orchestra.run.smithi157.stdout:Epoch : 1 2024-09-18T11:07:24.344 INFO:teuthology.orchestra.run.smithi157.stdout:Version : 2.06 2024-09-18T11:07:24.344 INFO:teuthology.orchestra.run.smithi157.stdout:Release : 82.el9 2024-09-18T11:07:24.344 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: x86_64 2024-09-18T11:07:24.344 INFO:teuthology.orchestra.run.smithi157.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-18T11:07:24.345 INFO:teuthology.orchestra.run.smithi157.stdout:Group : Unspecified 2024-09-18T11:07:24.345 INFO:teuthology.orchestra.run.smithi157.stdout:Size : 8274670 2024-09-18T11:07:24.345 INFO:teuthology.orchestra.run.smithi157.stdout:License : GPLv3+ 2024-09-18T11:07:24.345 INFO:teuthology.orchestra.run.smithi157.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-18T11:07:24.345 INFO:teuthology.orchestra.run.smithi157.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-18T11:07:24.345 INFO:teuthology.orchestra.run.smithi157.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-18T11:07:24.345 INFO:teuthology.orchestra.run.smithi157.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-18T11:07:24.345 INFO:teuthology.orchestra.run.smithi157.stdout:Packager : builder@centos.org 2024-09-18T11:07:24.345 INFO:teuthology.orchestra.run.smithi157.stdout:Vendor : CentOS 2024-09-18T11:07:24.345 INFO:teuthology.orchestra.run.smithi157.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-18T11:07:24.345 INFO:teuthology.orchestra.run.smithi157.stdout:Summary : Support tools for GRUB. 2024-09-18T11:07:24.345 INFO:teuthology.orchestra.run.smithi157.stdout:Description : 2024-09-18T11:07:24.345 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:07:24.346 INFO:teuthology.orchestra.run.smithi157.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-18T11:07:24.346 INFO:teuthology.orchestra.run.smithi157.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-18T11:07:24.346 INFO:teuthology.orchestra.run.smithi157.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-18T11:07:24.346 INFO:teuthology.orchestra.run.smithi157.stdout:hardware devices. 2024-09-18T11:07:24.346 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:07:24.346 INFO:teuthology.orchestra.run.smithi157.stdout:This subpackage provides tools for support of all platforms. 2024-09-18T11:07:24.347 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-18T11:07:24.347 INFO:teuthology.task.kernel:Updating grub on smithi157 to boot 5.14.0-508.el9.x86_64 2024-09-18T11:07:24.347 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-18T11:07:25.020 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2024-09-18T11:07:26.317 INFO:teuthology.orchestra.run.smithi157.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-18T11:07:26.356 INFO:teuthology.orchestra.run.smithi157.stderr:done 2024-09-18T11:07:26.358 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-09-18T11:07:26.374 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.1myBrvKGG3 2024-09-18T11:07:26.374 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1myBrvKGG3 2024-09-18T11:07:26.445 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.1myBrvKGG3 2024-09-18T11:07:26.581 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.1myBrvKGG3 is 6KB 2024-09-18T11:07:26.630 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.1myBrvKGG3 2024-09-18T11:07:26.645 DEBUG:teuthology.orchestra.run.smithi157:> sudo /bin/ls /boot/loader/entries || true 2024-09-18T11:07:26.708 INFO:teuthology.orchestra.run.smithi157.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-18T11:07:26.708 INFO:teuthology.orchestra.run.smithi157.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-18T11:07:26.708 INFO:teuthology.orchestra.run.smithi157.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-18T11:07:26.708 INFO:teuthology.orchestra.run.smithi157.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-18T11:07:26.711 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-18T11:07:26.870 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2024-09-18T11:07:56.901 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-18T11:07:56.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-18T11:07:56.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:08:15.297 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-18T11:08:24.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-18T11:08:24.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:08:27.357 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-18T11:08:39.369 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-18T11:08:39.370 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:08:42.429 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-18T11:08:57.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-18T11:08:57.442 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:09:15.965 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-18T11:09:33.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-18T11:09:33.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:09:34.259 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-09-18T11:09:34.674 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-18T11:09:34.674 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-18T11:09:34.675 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-09-18T11:09:34.689 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-508.el9.x86_64 2024-09-18T11:09:34.689 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-18T11:09:34.689 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T11:09:34.689 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-18T11:09:35.691 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-18T11:09:35.691 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T11:09:35.768 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2024-09-18T11:09:35.789 DEBUG:teuthology.parallel:result is None 2024-09-18T11:09:35.789 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-18T11:09:35.796 INFO:teuthology.task.internal:Creating test directory... 2024-09-18T11:09:35.797 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T11:09:35.812 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-18T11:09:35.862 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-18T11:09:35.922 INFO:teuthology.task.internal:Creating archive directory... 2024-09-18T11:09:35.922 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T11:09:35.960 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-18T11:09:35.986 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-18T11:09:35.986 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T11:09:36.040 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:09:36.049 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:09:36.051 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-18T11:09:36.059 INFO:teuthology.task.internal:Configuring sudo... 2024-09-18T11:09:36.059 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T11:09:36.119 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-18T11:09:36.127 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-18T11:09:36.127 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T11:09:36.178 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T11:09:36.262 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T11:09:36.343 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T11:09:36.384 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T11:09:36.428 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-18T11:09:36.428 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T11:09:36.495 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2024-09-18T11:09:36.594 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T11:09:37.000 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-18T11:09:37.008 INFO:teuthology.task.internal:Starting timer... 2024-09-18T11:09:37.009 INFO:teuthology.run_tasks:Running task pcp... 2024-09-18T11:09:37.155 INFO:teuthology.run_tasks:Running task selinux... 2024-09-18T11:09:37.266 DEBUG:teuthology.orchestra.run.smithi157:> sudo service auditd rotate 2024-09-18T11:09:37.313 INFO:teuthology.orchestra.run.smithi157.stdout:Rotating logs: 2024-09-18T11:09:37.315 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-18T11:09:37.315 DEBUG:teuthology.orchestra.run.smithi157:> /usr/sbin/getenforce 2024-09-18T11:09:37.351 INFO:teuthology.orchestra.run.smithi157.stdout:Permissive 2024-09-18T11:09:37.351 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi157.front.sepia.ceph.com': 'permissive'} 2024-09-18T11:09:37.351 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-18T11:09:37.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:09:37.417 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-18T11:09:37.417 DEBUG:teuthology.orchestra.run.smithi157:> sudo /usr/sbin/setenforce permissive 2024-09-18T11:09:37.488 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-18T11:09:37.497 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-18T11:09:37.500 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-18T11:09:37.612 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-18T11:09:37.623 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-18T11:09:37.623 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi157.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-18T11:13:20.222 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi157.front.sepia.ceph.com')] 2024-09-18T11:13:20.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-18T11:13:20.225 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:13:20.304 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-09-18T11:13:20.380 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-18T11:13:20.380 INFO:teuthology.run_tasks:Running task clock... 2024-09-18T11:13:20.391 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-18T11:13:20.392 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T11:13:20.392 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T11:13:20.449 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-18T11:13:20.463 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-18T11:13:20.488 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: ntpd: command not found 2024-09-18T11:13:20.499 INFO:teuthology.orchestra.run.smithi157.stdout:506 Cannot talk to daemon 2024-09-18T11:13:20.513 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-18T11:13:20.527 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-18T11:13:20.590 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-09-18T11:13:20.594 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T11:13:20.594 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-09-18T11:13:20.594 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T11:13:20.594 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T11:13:20.594 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T11:13:20.594 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T11:13:20.594 INFO:teuthology.run_tasks:Running task install... 2024-09-18T11:13:20.602 DEBUG:teuthology.task.install:project ceph 2024-09-18T11:13:20.602 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}} 2024-09-18T11:13:20.602 DEBUG:teuthology.task.install:config {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'} 2024-09-18T11:13:20.603 INFO:teuthology.task.install:Using flavor: default 2024-09-18T11:13:20.608 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-18T11:13:20.608 INFO:teuthology.task.install:extra packages: [] 2024-09-18T11:13:20.608 DEBUG:teuthology.orchestra.run.smithi157:> sudo dnf -y install 'dnf-command(copr)' 2024-09-18T11:13:21.206 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:45 ago on Wed 18 Sep 2024 11:12:36 AM UTC. 2024-09-18T11:13:21.293 INFO:teuthology.orchestra.run.smithi157.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-18T11:13:21.335 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:13:21.336 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-18T11:13:21.336 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:13:21.404 DEBUG:teuthology.orchestra.run.smithi157:> sudo dnf -y copr enable ceph/el9 2024-09-18T11:13:21.582 INFO:teuthology.orchestra.run.smithi157.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-18T11:13:21.582 INFO:teuthology.orchestra.run.smithi157.stderr:of the main distribution, and quality may vary. 2024-09-18T11:13:21.583 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-09-18T11:13:21.583 INFO:teuthology.orchestra.run.smithi157.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-18T11:13:21.583 INFO:teuthology.orchestra.run.smithi157.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-18T11:13:21.583 INFO:teuthology.orchestra.run.smithi157.stderr:, 2024-09-18T11:13:21.583 INFO:teuthology.orchestra.run.smithi157.stderr:and packages are not held to any quality or security level. 2024-09-18T11:13:21.583 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-09-18T11:13:21.583 INFO:teuthology.orchestra.run.smithi157.stderr:Please do not file bug reports about these packages in Fedora 2024-09-18T11:13:21.583 INFO:teuthology.orchestra.run.smithi157.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-18T11:13:21.656 INFO:teuthology.orchestra.run.smithi157.stdout:Repository successfully enabled. 2024-09-18T11:13:21.693 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'tag': None, 'wait_for_package': False} 2024-09-18T11:13:21.694 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:13:21.843 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/ 2024-09-18T11:13:21.844 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1645.g7061ea6c 2024-09-18T11:13:21.953 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-18T11:13:21.953 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-18T11:13:21.953 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-18T11:13:21.980 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-18T11:13:21.981 DEBUG:teuthology.orchestra.run.smithi157:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7061ea6ca50908d104e9f3979dc615e118d9e153/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-18T11:13:22.047 DEBUG:teuthology.orchestra.run.smithi157:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-18T11:13:22.123 DEBUG:teuthology.orchestra.run.smithi157:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-18T11:13:22.188 INFO:teuthology.orchestra.run.smithi157.stdout:check_obsoletes = 1 2024-09-18T11:13:22.190 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum clean all 2024-09-18T11:13:22.432 INFO:teuthology.orchestra.run.smithi157.stdout:45 files removed 2024-09-18T11:13:22.458 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-radosgw 2024-09-18T11:13:23.003 INFO:teuthology.orchestra.run.smithi157.stdout:ceph packages for x86_64 299 kB/s | 77 kB 00:00 2024-09-18T11:13:23.317 INFO:teuthology.orchestra.run.smithi157.stdout:ceph noarch packages 40 kB/s | 11 kB 00:00 2024-09-18T11:13:23.588 INFO:teuthology.orchestra.run.smithi157.stdout:ceph source packages 7.6 kB/s | 1.8 kB 00:00 2024-09-18T11:13:23.771 INFO:teuthology.orchestra.run.smithi157.stdout:Copr repo for el9 owned by ceph 51 kB/s | 7.6 kB 00:00 2024-09-18T11:13:24.263 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.3 MB 00:00 2024-09-18T11:13:25.923 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 35 MB/s | 20 MB 00:00 2024-09-18T11:13:30.833 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 6.5 MB 00:01 2024-09-18T11:13:32.529 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 19 kB 00:00 2024-09-18T11:13:33.671 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-18T11:13:39.110 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-18T11:13:41.427 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:13:41.455 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:13:41.455 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-18T11:13:41.455 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:13:41.455 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:13:41.455 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 14 M 2024-09-18T11:13:41.455 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrading: 2024-09-18T11:13:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: librados2 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 3.1 M 2024-09-18T11:13:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 2.9 M 2024-09-18T11:13:41.456 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-18T11:13:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 5.0 M 2024-09-18T11:13:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 22 M 2024-09-18T11:13:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 25 k 2024-09-18T11:13:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-18T11:13:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 637 k 2024-09-18T11:13:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-18T11:13:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-18T11:13:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-18T11:13:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 458 k 2024-09-18T11:13:41.457 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-18T11:13:41.457 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 4.9 M 2024-09-18T11:13:41.457 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-18T11:13:41.457 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-18T11:13:41.457 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 45 k 2024-09-18T11:13:41.457 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 120 k 2024-09-18T11:13:41.457 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 156 k 2024-09-18T11:13:41.457 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-18T11:13:41.457 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 320 k 2024-09-18T11:13:41.457 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 296 k 2024-09-18T11:13:41.457 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 99 k 2024-09-18T11:13:41.457 INFO:teuthology.orchestra.run.smithi157.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-18T11:13:41.458 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:13:41.458 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:13:41.458 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:13:41.458 INFO:teuthology.orchestra.run.smithi157.stdout:Install 22 Packages 2024-09-18T11:13:41.458 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrade 2 Packages 2024-09-18T11:13:41.458 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:13:41.458 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 58 M 2024-09-18T11:13:41.458 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:13:42.202 INFO:teuthology.orchestra.run.smithi157.stdout:(1/24): ceph-base-17.2.7-1645.g7061ea6c.el9.x86 12 MB/s | 5.0 MB 00:00 2024-09-18T11:13:42.253 INFO:teuthology.orchestra.run.smithi157.stdout:(2/24): ceph-selinux-17.2.7-1645.g7061ea6c.el9. 504 kB/s | 25 kB 00:00 2024-09-18T11:13:42.345 INFO:teuthology.orchestra.run.smithi157.stdout:(3/24): libcephfs2-17.2.7-1645.g7061ea6c.el9.x8 6.8 MB/s | 637 kB 00:00 2024-09-18T11:13:42.437 INFO:teuthology.orchestra.run.smithi157.stdout:(4/24): libradosstriper1-17.2.7-1645.g7061ea6c. 4.9 MB/s | 458 kB 00:00 2024-09-18T11:13:42.771 INFO:teuthology.orchestra.run.smithi157.stdout:(5/24): librgw2-17.2.7-1645.g7061ea6c.el9.x86_6 15 MB/s | 4.9 MB 00:00 2024-09-18T11:13:42.829 INFO:teuthology.orchestra.run.smithi157.stdout:(6/24): python3-ceph-argparse-17.2.7-1645.g7061 775 kB/s | 45 kB 00:00 2024-09-18T11:13:42.971 INFO:teuthology.orchestra.run.smithi157.stdout:(7/24): ceph-radosgw-17.2.7-1645.g7061ea6c.el9. 12 MB/s | 14 MB 00:01 2024-09-18T11:13:42.996 INFO:teuthology.orchestra.run.smithi157.stdout:(8/24): python3-ceph-common-17.2.7-1645.g7061ea 721 kB/s | 120 kB 00:00 2024-09-18T11:13:43.213 INFO:teuthology.orchestra.run.smithi157.stdout:(9/24): ceph-common-17.2.7-1645.g7061ea6c.el9.x 16 MB/s | 22 MB 00:01 2024-09-18T11:13:43.238 INFO:teuthology.orchestra.run.smithi157.stdout:(10/24): python3-cephfs-17.2.7-1645.g7061ea6c.e 585 kB/s | 156 kB 00:00 2024-09-18T11:13:43.272 INFO:teuthology.orchestra.run.smithi157.stdout:(11/24): python3-rados-17.2.7-1645.g7061ea6c.el 1.1 MB/s | 320 kB 00:00 2024-09-18T11:13:43.306 INFO:teuthology.orchestra.run.smithi157.stdout:(12/24): python3-rbd-17.2.7-1645.g7061ea6c.el9. 3.1 MB/s | 296 kB 00:00 2024-09-18T11:13:43.331 INFO:teuthology.orchestra.run.smithi157.stdout:(13/24): python3-rgw-17.2.7-1645.g7061ea6c.el9. 1.1 MB/s | 99 kB 00:00 2024-09-18T11:13:43.714 INFO:teuthology.orchestra.run.smithi157.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 118 kB/s | 45 kB 00:00 2024-09-18T11:13:43.881 INFO:teuthology.orchestra.run.smithi157.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 263 kB/s | 160 kB 00:00 2024-09-18T11:13:43.998 INFO:teuthology.orchestra.run.smithi157.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 362 kB/s | 42 kB 00:00 2024-09-18T11:13:44.024 INFO:teuthology.orchestra.run.smithi157.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 946 kB/s | 292 kB 00:00 2024-09-18T11:13:44.057 INFO:teuthology.orchestra.run.smithi157.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 912 kB/s | 662 kB 00:00 2024-09-18T11:13:44.107 INFO:teuthology.orchestra.run.smithi157.stdout:(19/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-18T11:13:44.133 INFO:teuthology.orchestra.run.smithi157.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2024-09-18T11:13:44.158 INFO:teuthology.orchestra.run.smithi157.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 741 kB/s | 67 kB 00:00 2024-09-18T11:13:44.250 INFO:teuthology.orchestra.run.smithi157.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-18T11:13:44.358 INFO:teuthology.orchestra.run.smithi157.stdout:(23/24): librados2-17.2.7-1645.g7061ea6c.el9.x8 14 MB/s | 3.1 MB 00:00 2024-09-18T11:13:44.417 INFO:teuthology.orchestra.run.smithi157.stdout:(24/24): librbd1-17.2.7-1645.g7061ea6c.el9.x86_ 11 MB/s | 2.9 MB 00:00 2024-09-18T11:13:44.422 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:13:44.422 INFO:teuthology.orchestra.run.smithi157.stdout:Total 20 MB/s | 58 MB 00:02 2024-09-18T11:13:44.488 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:13:44.522 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:13:44.522 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:13:45.174 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:13:45.174 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:13:46.404 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:13:46.790 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-18T11:13:46.861 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/26 2024-09-18T11:13:46.937 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/26 2024-09-18T11:13:47.016 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/26 2024-09-18T11:13:47.046 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/26 2024-09-18T11:13:47.117 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/26 2024-09-18T11:13:47.164 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-18T11:13:47.193 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-18T11:13:47.637 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-18T11:13:47.674 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 8/26 2024-09-18T11:13:47.737 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 8/26 2024-09-18T11:13:47.774 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-18T11:13:47.816 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-argparse-2:17.2.7-1645.g7061ea6c.el9. 10/26 2024-09-18T11:13:47.853 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 11/26 2024-09-18T11:13:47.925 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 12/26 2024-09-18T11:13:47.951 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_6 13/26 2024-09-18T11:13:48.000 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_6 13/26 2024-09-18T11:13:48.041 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-18T11:13:48.093 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-18T11:13:48.134 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-18T11:13:48.435 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-18T11:13:48.480 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 18/26 2024-09-18T11:13:48.553 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 18/26 2024-09-18T11:13:48.628 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 19/26 2024-09-18T11:13:48.658 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-common-2:17.2.7-1645.g7061ea6c.el9.x8 20/26 2024-09-18T11:13:51.094 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 21/26 2024-09-18T11:13:51.129 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 21/26 2024-09-18T11:13:51.646 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 21/26 2024-09-18T11:13:51.691 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 22/26 2024-09-18T11:13:51.789 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 22/26 2024-09-18T11:13:51.789 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-18T11:13:51.789 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-18T11:13:51.789 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:13:51.825 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 23/26 2024-09-18T11:14:01.870 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 23/26 2024-09-18T11:14:01.871 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /sys 2024-09-18T11:14:01.871 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /proc 2024-09-18T11:14:01.871 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /mnt 2024-09-18T11:14:01.871 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /var/tmp 2024-09-18T11:14:01.871 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /home 2024-09-18T11:14:01.871 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /root 2024-09-18T11:14:01.871 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /tmp 2024-09-18T11:14:01.871 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:03.217 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 24/26 2024-09-18T11:14:03.279 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 24/26 2024-09-18T11:14:03.279 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T11:14:03.279 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-18T11:14:03.279 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T11:14:03.279 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T11:14:03.280 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:03.281 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-18T11:14:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-18T11:14:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-18T11:14:04.611 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-18T11:14:04.612 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/26 2024-09-18T11:14:04.612 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/26 2024-09-18T11:14:04.612 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/26 2024-09-18T11:14:04.612 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/26 2024-09-18T11:14:04.612 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 5/26 2024-09-18T11:14:04.612 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_6 6/26 2024-09-18T11:14:04.612 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 7/26 2024-09-18T11:14:04.612 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1645.g7061ea6c.el9. 8/26 2024-09-18T11:14:04.612 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-common-2:17.2.7-1645.g7061ea6c.el9.x8 9/26 2024-09-18T11:14:04.613 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 10/26 2024-09-18T11:14:04.613 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 11/26 2024-09-18T11:14:04.613 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 12/26 2024-09-18T11:14:04.613 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 13/26 2024-09-18T11:14:04.613 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-18T11:14:04.613 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-18T11:14:04.613 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-18T11:14:04.613 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-18T11:14:04.614 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-18T11:14:04.614 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-18T11:14:04.614 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-18T11:14:04.614 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-18T11:14:04.614 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-18T11:14:04.614 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 23/26 2024-09-18T11:14:04.614 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-18T11:14:04.614 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 25/26 2024-09-18T11:14:05.662 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-18T11:14:05.662 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:05.662 INFO:teuthology.orchestra.run.smithi157.stdout:Upgraded: 2024-09-18T11:14:05.662 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:05.662 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:05.662 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:14:05.662 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:05.662 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:05.662 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:05.662 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:05.663 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-18T11:14:05.663 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:05.663 INFO:teuthology.orchestra.run.smithi157.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-18T11:14:05.663 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-18T11:14:05.663 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-18T11:14:05.663 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:05.663 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-18T11:14:05.663 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:05.663 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-18T11:14:05.663 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-18T11:14:05.663 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:05.663 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:05.664 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:05.664 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-18T11:14:05.664 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:05.664 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:05.664 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:05.664 INFO:teuthology.orchestra.run.smithi157.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-18T11:14:05.664 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:05.664 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:14:06.024 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-test 2024-09-18T11:14:06.610 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:27 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:14:06.741 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:14:06.742 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:06.742 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-18T11:14:06.742 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:06.742 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:14:06.742 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 48 M 2024-09-18T11:14:06.742 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-18T11:14:06.742 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 163 k 2024-09-18T11:14:06.742 INFO:teuthology.orchestra.run.smithi157.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-18T11:14:06.742 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-18T11:14:06.743 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:06.743 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:14:06.743 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:06.743 INFO:teuthology.orchestra.run.smithi157.stdout:Install 4 Packages 2024-09-18T11:14:06.743 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:06.743 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 48 M 2024-09-18T11:14:06.744 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 193 M 2024-09-18T11:14:06.744 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:14:07.083 INFO:teuthology.orchestra.run.smithi157.stdout:(1/4): libcephsqlite-17.2.7-1645.g7061ea6c.el9. 757 kB/s | 163 kB 00:00 2024-09-18T11:14:07.117 INFO:teuthology.orchestra.run.smithi157.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-18T11:14:07.142 INFO:teuthology.orchestra.run.smithi157.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-09-18T11:14:08.368 INFO:teuthology.orchestra.run.smithi157.stdout:(4/4): ceph-test-17.2.7-1645.g7061ea6c.el9.x86_ 32 MB/s | 48 MB 00:01 2024-09-18T11:14:08.369 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:14:08.370 INFO:teuthology.orchestra.run.smithi157.stdout:Total 30 MB/s | 48 MB 00:01 2024-09-18T11:14:08.398 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:14:08.417 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:14:08.417 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:14:08.783 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:14:08.784 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:14:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:14:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-18T11:14:09.458 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-18T11:14:09.486 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/4 2024-09-18T11:14:13.937 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/4 2024-09-18T11:14:14.041 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/4 2024-09-18T11:14:16.143 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/4 2024-09-18T11:14:16.143 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/4 2024-09-18T11:14:16.143 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/4 2024-09-18T11:14:16.143 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-18T11:14:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-18T11:14:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:16.455 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:14:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-18T11:14:16.456 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-18T11:14:16.456 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:16.456 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:14:16.712 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph 2024-09-18T11:14:17.288 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:38 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:14:17.426 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:14:17.429 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:17.429 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-18T11:14:17.429 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:17.429 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:14:17.429 INFO:teuthology.orchestra.run.smithi157.stdout: ceph x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 6.5 k 2024-09-18T11:14:17.429 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-18T11:14:17.429 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 2.1 M 2024-09-18T11:14:17.429 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 1.4 M 2024-09-18T11:14:17.429 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 240 k 2024-09-18T11:14:17.429 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 3.9 M 2024-09-18T11:14:17.430 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 15 M 2024-09-18T11:14:17.430 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-18T11:14:17.430 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-18T11:14:17.430 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-18T11:14:17.430 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-18T11:14:17.430 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-18T11:14:17.430 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-18T11:14:17.430 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-18T11:14:17.430 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections 2024-09-18T11:14:17.430 INFO:teuthology.orchestra.run.smithi157.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-18T11:14:17.430 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-18T11:14:17.431 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-18T11:14:17.431 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-18T11:14:17.431 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-18T11:14:17.431 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-18T11:14:17.431 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-18T11:14:17.431 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-18T11:14:17.431 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-18T11:14:17.431 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-18T11:14:17.431 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-18T11:14:17.431 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-18T11:14:17.431 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-18T11:14:17.431 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-18T11:14:17.431 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-18T11:14:17.432 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-18T11:14:17.432 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-18T11:14:17.432 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-18T11:14:17.432 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-18T11:14:17.432 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-18T11:14:17.432 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-18T11:14:17.432 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:17.432 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:14:17.432 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:17.432 INFO:teuthology.orchestra.run.smithi157.stdout:Install 34 Packages 2024-09-18T11:14:17.432 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:17.433 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 27 M 2024-09-18T11:14:17.433 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 96 M 2024-09-18T11:14:17.433 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:14:18.177 INFO:teuthology.orchestra.run.smithi157.stdout:(1/34): ceph-17.2.7-1645.g7061ea6c.el9.x86_64.r 51 kB/s | 6.5 kB 00:00 2024-09-18T11:14:18.394 INFO:teuthology.orchestra.run.smithi157.stdout:(2/34): ceph-mds-17.2.7-1645.g7061ea6c.el9.x86_ 6.0 MB/s | 2.1 MB 00:00 2024-09-18T11:14:18.436 INFO:teuthology.orchestra.run.smithi157.stdout:(3/34): ceph-mgr-17.2.7-1645.g7061ea6c.el9.x86_ 3.6 MB/s | 1.4 MB 00:00 2024-09-18T11:14:18.519 INFO:teuthology.orchestra.run.smithi157.stdout:(4/34): ceph-mon-17.2.7-1645.g7061ea6c.el9.x86_ 11 MB/s | 3.9 MB 00:00 2024-09-18T11:14:18.544 INFO:teuthology.orchestra.run.smithi157.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1645.g7061 2.2 MB/s | 240 kB 00:00 2024-09-18T11:14:18.670 INFO:teuthology.orchestra.run.smithi157.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-09-18T11:14:18.711 INFO:teuthology.orchestra.run.smithi157.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-18T11:14:18.762 INFO:teuthology.orchestra.run.smithi157.stdout:(8/34): python3-cryptography-36.0.1-4.el9.x86_6 5.6 MB/s | 1.2 MB 00:00 2024-09-18T11:14:18.787 INFO:teuthology.orchestra.run.smithi157.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 1.8 MB/s | 135 kB 00:00 2024-09-18T11:14:18.812 INFO:teuthology.orchestra.run.smithi157.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-18T11:14:18.838 INFO:teuthology.orchestra.run.smithi157.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-18T11:14:18.871 INFO:teuthology.orchestra.run.smithi157.stdout:(12/34): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-18T11:14:18.897 INFO:teuthology.orchestra.run.smithi157.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 592 kB/s | 35 kB 00:00 2024-09-18T11:14:18.922 INFO:teuthology.orchestra.run.smithi157.stdout:(14/34): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-18T11:14:18.947 INFO:teuthology.orchestra.run.smithi157.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-18T11:14:18.972 INFO:teuthology.orchestra.run.smithi157.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 830 kB/s | 42 kB 00:00 2024-09-18T11:14:19.014 INFO:teuthology.orchestra.run.smithi157.stdout:(17/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 649 kB/s | 43 kB 00:00 2024-09-18T11:14:19.048 INFO:teuthology.orchestra.run.smithi157.stdout:(18/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-09-18T11:14:19.073 INFO:teuthology.orchestra.run.smithi157.stdout:(19/34): python3-cherrypy-18.6.1-2.el9.noarch.r 6.2 MB/s | 358 kB 00:00 2024-09-18T11:14:19.099 INFO:teuthology.orchestra.run.smithi157.stdout:(20/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-09-18T11:14:19.265 INFO:teuthology.orchestra.run.smithi157.stdout:(21/34): ceph-osd-17.2.7-1645.g7061ea6c.el9.x86 18 MB/s | 15 MB 00:00 2024-09-18T11:14:19.291 INFO:teuthology.orchestra.run.smithi157.stdout:(22/34): python3-jaraco-classes-3.2.1-5.el9.noa 92 kB/s | 18 kB 00:00 2024-09-18T11:14:19.316 INFO:teuthology.orchestra.run.smithi157.stdout:(23/34): python3-jaraco-collections-3.0.0-8.el9 107 kB/s | 23 kB 00:00 2024-09-18T11:14:19.341 INFO:teuthology.orchestra.run.smithi157.stdout:(24/34): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-09-18T11:14:19.368 INFO:teuthology.orchestra.run.smithi157.stdout:(25/34): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-09-18T11:14:19.392 INFO:teuthology.orchestra.run.smithi157.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-18T11:14:19.409 INFO:teuthology.orchestra.run.smithi157.stdout:(27/34): python3-logutils-0.3.5-21.el9.noarch.r 501 kB/s | 46 kB 00:00 2024-09-18T11:14:19.425 INFO:teuthology.orchestra.run.smithi157.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 491 kB/s | 16 kB 00:00 2024-09-18T11:14:19.454 INFO:teuthology.orchestra.run.smithi157.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.9 MB/s | 90 kB 00:00 2024-09-18T11:14:19.479 INFO:teuthology.orchestra.run.smithi157.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 666 kB/s | 36 kB 00:00 2024-09-18T11:14:19.513 INFO:teuthology.orchestra.run.smithi157.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-18T11:14:19.547 INFO:teuthology.orchestra.run.smithi157.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-18T11:14:19.572 INFO:teuthology.orchestra.run.smithi157.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 343 kB/s | 20 kB 00:00 2024-09-18T11:14:19.614 INFO:teuthology.orchestra.run.smithi157.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-09-18T11:14:19.619 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:14:19.619 INFO:teuthology.orchestra.run.smithi157.stdout:Total 12 MB/s | 27 MB 00:02 2024-09-18T11:14:19.749 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:14:19.786 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:14:19.786 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:14:20.219 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:14:20.220 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:14:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:14:21.550 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-18T11:14:21.582 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-18T11:14:21.623 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-18T11:14:21.716 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-18T11:14:21.794 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-18T11:14:21.845 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-18T11:14:21.878 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-18T11:14:21.908 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-18T11:14:21.943 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-18T11:14:21.976 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-18T11:14:22.017 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-18T11:14:22.053 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-18T11:14:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-18T11:14:22.134 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-18T11:14:22.177 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-18T11:14:22.264 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-18T11:14:22.410 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-18T11:14:22.491 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-18T11:14:22.544 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-18T11:14:22.583 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-18T11:14:22.632 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-18T11:14:22.695 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-18T11:14:22.888 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-18T11:14:22.945 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-18T11:14:23.024 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-18T11:14:23.112 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-18T11:14:23.170 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-18T11:14:23.293 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-18T11:14:23.449 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1645.g7061ea6c.el9. 29/34 2024-09-18T11:14:23.481 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 30/34 2024-09-18T11:14:23.507 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 30/34 2024-09-18T11:14:23.507 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T11:14:23.507 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-18T11:14:23.507 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T11:14:23.507 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T11:14:23.507 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:24.814 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 31/34 2024-09-18T11:14:24.842 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 31/34 2024-09-18T11:14:24.842 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T11:14:24.842 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-18T11:14:24.842 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T11:14:24.842 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T11:14:24.842 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:25.224 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 32/34 2024-09-18T11:14:25.248 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 32/34 2024-09-18T11:14:25.248 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T11:14:25.248 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-18T11:14:25.248 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T11:14:25.249 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T11:14:25.249 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:25.466 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 33/34 2024-09-18T11:14:25.489 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 33/34 2024-09-18T11:14:25.489 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T11:14:25.489 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-18T11:14:25.490 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T11:14:25.490 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T11:14:25.490 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:25.554 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 34/34 2024-09-18T11:14:27.404 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 34/34 2024-09-18T11:14:27.405 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/34 2024-09-18T11:14:27.405 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/34 2024-09-18T11:14:27.405 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/34 2024-09-18T11:14:27.405 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/34 2024-09-18T11:14:27.405 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 5/34 2024-09-18T11:14:27.405 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1645.g7061ea6c.el9. 6/34 2024-09-18T11:14:27.405 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-18T11:14:27.405 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-18T11:14:27.405 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-18T11:14:27.405 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-18T11:14:27.405 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-18T11:14:27.405 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-18T11:14:27.405 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-18T11:14:27.406 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-18T11:14:27.406 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-18T11:14:27.406 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-18T11:14:27.406 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-18T11:14:27.406 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-18T11:14:27.406 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-18T11:14:27.406 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-18T11:14:27.406 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-18T11:14:27.406 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-18T11:14:27.406 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-18T11:14:27.406 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-18T11:14:27.406 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-18T11:14:27.406 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-18T11:14:27.406 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-18T11:14:27.406 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-18T11:14:27.407 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-18T11:14:27.407 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-18T11:14:27.407 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-18T11:14:27.407 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-18T11:14:27.407 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-18T11:14:28.426 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-18T11:14:28.426 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:28.426 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:14:28.426 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:28.426 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:28.426 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:28.426 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:14:28.426 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:28.427 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:28.427 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-18T11:14:28.427 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-18T11:14:28.427 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-18T11:14:28.427 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-18T11:14:28.427 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-18T11:14:28.427 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-18T11:14:28.427 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-18T11:14:28.427 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-18T11:14:28.427 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-18T11:14:28.427 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-18T11:14:28.427 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-18T11:14:28.427 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-18T11:14:28.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-18T11:14:28.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-18T11:14:28.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-18T11:14:28.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-18T11:14:28.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply-3.11-14.el9.noarch 2024-09-18T11:14:28.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-18T11:14:28.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-18T11:14:28.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-18T11:14:28.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-18T11:14:28.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-18T11:14:28.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-18T11:14:28.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-18T11:14:28.429 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-18T11:14:28.429 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-18T11:14:28.429 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-18T11:14:28.429 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-18T11:14:28.429 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:28.429 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:14:28.793 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-base 2024-09-18T11:14:29.382 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:50 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:14:29.473 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:14:29.515 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:14:29.516 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-18T11:14:29.516 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:14:29.589 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install cephadm 2024-09-18T11:14:30.166 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:51 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:14:30.296 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:14:30.296 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:30.296 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-18T11:14:30.296 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:30.297 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:14:30.297 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 83 k 2024-09-18T11:14:30.297 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:30.297 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:14:30.297 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:30.297 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-18T11:14:30.297 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:30.297 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 83 k 2024-09-18T11:14:30.297 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 375 k 2024-09-18T11:14:30.297 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:14:30.491 INFO:teuthology.orchestra.run.smithi157.stdout:cephadm-17.2.7-1645.g7061ea6c.el9.noarch.rpm 427 kB/s | 83 kB 00:00 2024-09-18T11:14:30.492 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:14:30.492 INFO:teuthology.orchestra.run.smithi157.stdout:Total 423 kB/s | 83 kB 00:00 2024-09-18T11:14:30.492 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:14:30.498 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:14:30.498 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:14:30.499 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:14:30.500 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:14:30.570 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:14:31.007 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T11:14:31.111 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T11:14:31.452 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T11:14:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T11:14:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:31.711 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:14:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:14:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:31.711 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:14:31.911 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-immutable-object-cache 2024-09-18T11:14:32.500 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:53 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:14:32.637 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:14:32.637 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:32.638 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-09-18T11:14:32.638 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:32.638 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:14:32.638 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 137 k 2024-09-18T11:14:32.638 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:32.638 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:14:32.638 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:32.638 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-18T11:14:32.638 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:32.638 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 137 k 2024-09-18T11:14:32.638 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 379 k 2024-09-18T11:14:32.638 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:14:32.831 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-immutable-object-cache-17.2.7-1645.g7061ea 712 kB/s | 137 kB 00:00 2024-09-18T11:14:32.832 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:14:32.832 INFO:teuthology.orchestra.run.smithi157.stdout:Total 707 kB/s | 137 kB 00:00 2024-09-18T11:14:32.832 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:14:32.839 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:14:32.840 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:14:32.896 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:14:32.896 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:14:33.031 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:14:33.092 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c. 1/1 2024-09-18T11:14:33.116 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c. 1/1 2024-09-18T11:14:33.116 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T11:14:33.116 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-18T11:14:33.116 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:34.018 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c. 1/1 2024-09-18T11:14:34.018 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:34.018 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:14:34.018 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:14:34.018 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:34.018 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:14:34.232 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr 2024-09-18T11:14:34.821 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:55 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:14:34.914 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:14:34.958 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:14:34.959 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-18T11:14:34.959 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:14:35.023 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-dashboard 2024-09-18T11:14:35.599 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:56 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:14:35.730 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:14:35.731 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-18T11:14:35.731 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:35.731 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:14:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 3.4 M 2024-09-18T11:14:35.731 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-18T11:14:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 24 k 2024-09-18T11:14:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 15 k 2024-09-18T11:14:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-18T11:14:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-18T11:14:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-18T11:14:35.732 INFO:teuthology.orchestra.run.smithi157.stdout:Installing weak dependencies: 2024-09-18T11:14:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-18T11:14:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:35.732 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:14:35.732 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:35.732 INFO:teuthology.orchestra.run.smithi157.stdout:Install 7 Packages 2024-09-18T11:14:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:35.733 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 3.7 M 2024-09-18T11:14:35.733 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 80 M 2024-09-18T11:14:35.733 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:14:35.951 INFO:teuthology.orchestra.run.smithi157.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1645.g706 156 kB/s | 24 kB 00:00 2024-09-18T11:14:35.968 INFO:teuthology.orchestra.run.smithi157.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1645.g7061 84 kB/s | 15 kB 00:00 2024-09-18T11:14:36.002 INFO:teuthology.orchestra.run.smithi157.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 180 kB/s | 9.0 kB 00:00 2024-09-18T11:14:36.019 INFO:teuthology.orchestra.run.smithi157.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 2.0 MB/s | 31 kB 00:00 2024-09-18T11:14:36.036 INFO:teuthology.orchestra.run.smithi157.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 608 kB/s | 41 kB 00:00 2024-09-18T11:14:36.061 INFO:teuthology.orchestra.run.smithi157.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-18T11:14:36.186 INFO:teuthology.orchestra.run.smithi157.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1645.g7061ea6c 8.6 MB/s | 3.4 MB 00:00 2024-09-18T11:14:36.189 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:14:36.190 INFO:teuthology.orchestra.run.smithi157.stdout:Total 8.0 MB/s | 3.7 MB 00:00 2024-09-18T11:14:36.190 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:14:36.198 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:14:36.198 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:14:36.283 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:14:36.296 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:14:36.585 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:14:36.645 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-18T11:14:36.688 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-18T11:14:36.751 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-18T11:14:36.784 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-18T11:14:36.812 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1645.g7061ea6c.el9.n 5/7 2024-09-18T11:14:37.808 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1645.g7061ea6c.el9. 6/7 2024-09-18T11:14:37.841 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarc 7/7 2024-09-18T11:14:38.876 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarc 7/7 2024-09-18T11:14:38.876 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1645.g7061ea6c.el9. 1/7 2024-09-18T11:14:38.876 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarc 2/7 2024-09-18T11:14:38.876 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1645.g7061ea6c.el9.n 3/7 2024-09-18T11:14:38.876 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-18T11:14:38.876 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-18T11:14:38.876 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-18T11:14:39.161 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-18T11:14:39.161 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:39.161 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:14:39.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:14:39.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:14:39.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:14:39.161 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-18T11:14:39.161 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-18T11:14:39.161 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-18T11:14:39.161 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-18T11:14:39.161 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:39.162 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:14:39.398 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-18T11:14:39.977 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:00 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:14:40.108 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:14:40.109 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================= 2024-09-18T11:14:40.110 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-18T11:14:40.110 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================= 2024-09-18T11:14:40.110 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:14:40.110 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 7.4 M 2024-09-18T11:14:40.110 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-18T11:14:40.110 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-18T11:14:40.110 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-18T11:14:40.110 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-18T11:14:40.110 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-18T11:14:40.110 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-18T11:14:40.110 INFO:teuthology.orchestra.run.smithi157.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-18T11:14:40.110 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-18T11:14:40.110 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-18T11:14:40.111 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-18T11:14:40.111 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-18T11:14:40.111 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-18T11:14:40.111 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:40.111 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:14:40.111 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================= 2024-09-18T11:14:40.111 INFO:teuthology.orchestra.run.smithi157.stdout:Install 12 Packages 2024-09-18T11:14:40.111 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:40.112 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 38 M 2024-09-18T11:14:40.112 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 212 M 2024-09-18T11:14:40.112 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:14:40.777 INFO:teuthology.orchestra.run.smithi157.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 913 kB/s | 191 kB 00:00 2024-09-18T11:14:40.902 INFO:teuthology.orchestra.run.smithi157.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.3 MB/s | 801 kB 00:00 2024-09-18T11:14:40.927 INFO:teuthology.orchestra.run.smithi157.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 218 kB/s | 33 kB 00:00 2024-09-18T11:14:40.995 INFO:teuthology.orchestra.run.smithi157.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 272 kB/s | 18 kB 00:00 2024-09-18T11:14:41.070 INFO:teuthology.orchestra.run.smithi157.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 515 kB/s | 38 kB 00:00 2024-09-18T11:14:41.128 INFO:teuthology.orchestra.run.smithi157.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-09-18T11:14:41.204 INFO:teuthology.orchestra.run.smithi157.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-09-18T11:14:41.288 INFO:teuthology.orchestra.run.smithi157.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 23 MB/s | 4.9 MB 00:00 2024-09-18T11:14:41.388 INFO:teuthology.orchestra.run.smithi157.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 5.1 MB 00:00 2024-09-18T11:14:41.413 INFO:teuthology.orchestra.run.smithi157.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.0 MB/s | 250 kB 00:00 2024-09-18T11:14:41.655 INFO:teuthology.orchestra.run.smithi157.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 61 MB/s | 16 MB 00:00 2024-09-18T11:14:41.972 INFO:teuthology.orchestra.run.smithi157.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-1 5.3 MB/s | 7.4 MB 00:01 2024-09-18T11:14:41.973 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:14:41.973 INFO:teuthology.orchestra.run.smithi157.stdout:Total 21 MB/s | 38 MB 00:01 2024-09-18T11:14:42.325 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:14:42.352 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:14:42.352 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:14:42.861 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:14:42.862 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:14:43.624 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:14:43.707 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-18T11:14:43.745 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-18T11:14:43.853 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-18T11:14:43.888 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-18T11:14:44.073 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-18T11:14:44.103 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-18T11:14:44.180 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-18T11:14:44.862 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-18T11:14:44.983 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-18T11:14:46.282 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T11:14:47.326 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-18T11:14:47.352 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061e 12/12 2024-09-18T11:14:49.912 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061e 12/12 2024-09-18T11:14:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061e 1/12 2024-09-18T11:14:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-18T11:14:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-18T11:14:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-18T11:14:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-18T11:14:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-18T11:14:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-18T11:14:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-18T11:14:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-18T11:14:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T11:14:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-18T11:14:50.351 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-18T11:14:50.351 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:50.351 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:14:50.351 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:14:50.351 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-18T11:14:50.351 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-18T11:14:50.351 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-18T11:14:50.352 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-18T11:14:50.352 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-18T11:14:50.352 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-18T11:14:50.352 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-18T11:14:50.352 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-18T11:14:50.352 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-18T11:14:50.352 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-18T11:14:50.352 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-18T11:14:50.352 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:50.352 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:14:50.744 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-rook 2024-09-18T11:14:51.360 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:12 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:14:51.502 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:14:51.504 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-18T11:14:51.504 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:51.504 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:14:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 47 k 2024-09-18T11:14:51.504 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-18T11:14:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-18T11:14:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-18T11:14:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-18T11:14:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-18T11:14:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-18T11:14:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-18T11:14:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-18T11:14:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-18T11:14:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-18T11:14:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-18T11:14:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-18T11:14:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-18T11:14:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:51.505 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:14:51.505 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:51.505 INFO:teuthology.orchestra.run.smithi157.stdout:Install 13 Packages 2024-09-18T11:14:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:51.506 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 2.2 M 2024-09-18T11:14:51.506 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 26 M 2024-09-18T11:14:51.506 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:14:51.986 INFO:teuthology.orchestra.run.smithi157.stdout:(1/13): ceph-mgr-rook-17.2.7-1645.g7061ea6c.el9 299 kB/s | 47 kB 00:00 2024-09-18T11:14:52.348 INFO:teuthology.orchestra.run.smithi157.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 37 kB/s | 19 kB 00:00 2024-09-18T11:14:52.378 INFO:teuthology.orchestra.run.smithi157.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 46 kB/s | 26 kB 00:00 2024-09-18T11:14:52.729 INFO:teuthology.orchestra.run.smithi157.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 299 kB/s | 222 kB 00:00 2024-09-18T11:14:53.079 INFO:teuthology.orchestra.run.smithi157.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 216 kB/s | 159 kB 00:00 2024-09-18T11:14:53.129 INFO:teuthology.orchestra.run.smithi157.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 647 kB/s | 32 kB 00:00 2024-09-18T11:14:53.154 INFO:teuthology.orchestra.run.smithi157.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 359 kB/s | 279 kB 00:00 2024-09-18T11:14:53.179 INFO:teuthology.orchestra.run.smithi157.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 281 kB/s | 14 kB 00:00 2024-09-18T11:14:53.238 INFO:teuthology.orchestra.run.smithi157.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 2.6 MB/s | 220 kB 00:00 2024-09-18T11:14:53.280 INFO:teuthology.orchestra.run.smithi157.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 11 MB/s | 1.0 MB 00:00 2024-09-18T11:14:53.306 INFO:teuthology.orchestra.run.smithi157.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 883 kB/s | 59 kB 00:00 2024-09-18T11:14:53.331 INFO:teuthology.orchestra.run.smithi157.stdout:(12/13): python3-requests-oauthlib-1.3.0-12.el9 89 kB/s | 54 kB 00:00 2024-09-18T11:14:53.356 INFO:teuthology.orchestra.run.smithi157.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-09-18T11:14:53.359 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:14:53.359 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.2 MB/s | 2.2 MB 00:01 2024-09-18T11:14:53.425 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:14:53.441 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:14:53.441 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:14:53.530 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:14:53.532 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:14:54.041 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:14:54.211 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-18T11:14:54.274 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-18T11:14:54.337 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-18T11:14:54.375 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-18T11:14:54.416 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-18T11:14:54.535 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-18T11:14:54.636 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-18T11:14:54.693 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-18T11:14:55.414 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-18T11:14:55.465 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-18T11:14:55.499 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-18T11:14:55.544 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-18T11:14:55.568 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 13/13 2024-09-18T11:14:56.353 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 13/13 2024-09-18T11:14:56.353 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 1/13 2024-09-18T11:14:56.353 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-18T11:14:56.353 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-18T11:14:56.353 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-18T11:14:56.353 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-18T11:14:56.353 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-18T11:14:56.353 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-18T11:14:56.354 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-18T11:14:56.354 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-18T11:14:56.354 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-18T11:14:56.354 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-18T11:14:56.354 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-18T11:14:56.681 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-18T11:14:56.682 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:56.682 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:14:56.682 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:14:56.682 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-18T11:14:56.682 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-18T11:14:56.682 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-18T11:14:56.682 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-18T11:14:56.682 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-18T11:14:56.682 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-18T11:14:56.682 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-18T11:14:56.682 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-18T11:14:56.682 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-18T11:14:56.682 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-18T11:14:56.683 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-18T11:14:56.683 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-18T11:14:56.683 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:56.683 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:14:56.945 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-cephadm 2024-09-18T11:14:57.530 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:18 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:14:57.662 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:14:57.662 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:57.662 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-18T11:14:57.662 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:57.663 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:14:57.663 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 124 k 2024-09-18T11:14:57.663 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-18T11:14:57.663 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-18T11:14:57.663 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-18T11:14:57.664 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-18T11:14:57.664 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-18T11:14:57.664 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-18T11:14:57.664 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:57.664 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:14:57.664 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:14:57.664 INFO:teuthology.orchestra.run.smithi157.stdout:Install 6 Packages 2024-09-18T11:14:57.664 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:14:57.665 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 7.0 M 2024-09-18T11:14:57.665 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 33 M 2024-09-18T11:14:57.665 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:14:58.008 INFO:teuthology.orchestra.run.smithi157.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-09-18T11:14:58.033 INFO:teuthology.orchestra.run.smithi157.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1645.g7061ea6c.e 588 kB/s | 124 kB 00:00 2024-09-18T11:14:58.091 INFO:teuthology.orchestra.run.smithi157.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 990 kB/s | 58 kB 00:00 2024-09-18T11:14:58.116 INFO:teuthology.orchestra.run.smithi157.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-18T11:14:58.141 INFO:teuthology.orchestra.run.smithi157.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.0 MB/s | 548 kB 00:00 2024-09-18T11:14:58.283 INFO:teuthology.orchestra.run.smithi157.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-09-18T11:14:58.286 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:14:58.286 INFO:teuthology.orchestra.run.smithi157.stdout:Total 11 MB/s | 7.0 MB 00:00 2024-09-18T11:14:58.356 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:14:58.367 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:14:58.368 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:14:58.466 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:14:58.466 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:14:58.769 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:14:58.941 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-18T11:14:58.994 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-18T11:14:59.525 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-18T11:14:59.614 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-18T11:14:59.677 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-18T11:14:59.706 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 6/6 2024-09-18T11:15:00.467 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 6/6 2024-09-18T11:15:00.467 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/6 2024-09-18T11:15:00.467 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-18T11:15:00.467 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-18T11:15:00.467 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-18T11:15:00.467 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-18T11:15:00.739 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-18T11:15:00.740 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:00.740 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:15:00.740 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:15:00.740 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-18T11:15:00.740 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-18T11:15:00.740 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-18T11:15:00.740 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-18T11:15:00.740 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-18T11:15:00.740 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:00.740 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:15:00.981 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-fuse 2024-09-18T11:15:01.555 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:22 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:15:01.688 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:15:01.688 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:01.688 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-18T11:15:01.688 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:01.688 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:15:01.688 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 813 k 2024-09-18T11:15:01.689 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-18T11:15:01.689 INFO:teuthology.orchestra.run.smithi157.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-18T11:15:01.689 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:01.689 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:15:01.689 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:01.689 INFO:teuthology.orchestra.run.smithi157.stdout:Install 2 Packages 2024-09-18T11:15:01.689 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:01.690 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 893 k 2024-09-18T11:15:01.690 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 2.6 M 2024-09-18T11:15:01.690 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:15:02.194 INFO:teuthology.orchestra.run.smithi157.stdout:(1/2): ceph-fuse-17.2.7-1645.g7061ea6c.el9.x86_ 2.8 MB/s | 813 kB 00:00 2024-09-18T11:15:02.554 INFO:teuthology.orchestra.run.smithi157.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 125 kB/s | 80 kB 00:00 2024-09-18T11:15:02.554 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:15:02.554 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.0 MB/s | 893 kB 00:00 2024-09-18T11:15:02.564 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:15:02.586 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:15:02.586 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:15:02.665 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:15:02.666 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:15:02.854 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:15:03.001 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-18T11:15:03.075 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/2 2024-09-18T11:15:03.731 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/2 2024-09-18T11:15:03.731 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/2 2024-09-18T11:15:04.030 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-18T11:15:04.030 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:04.030 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:15:04.030 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-18T11:15:04.031 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:04.031 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:15:04.340 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-volume 2024-09-18T11:15:04.932 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:25 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:15:05.068 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:15:05.069 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:05.069 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-18T11:15:05.069 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:05.069 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:15:05.069 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 270 k 2024-09-18T11:15:05.069 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:05.069 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:15:05.070 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:05.070 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-18T11:15:05.070 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:05.070 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 270 k 2024-09-18T11:15:05.070 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 1.3 M 2024-09-18T11:15:05.070 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:15:05.299 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-volume-17.2.7-1645.g7061ea6c.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-18T11:15:05.299 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:15:05.299 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-18T11:15:05.299 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:15:05.308 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:15:05.308 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:15:05.332 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:15:05.333 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:15:05.548 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:15:05.617 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T11:15:05.633 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T11:15:05.633 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T11:15:05.633 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-18T11:15:05.633 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:06.672 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T11:15:06.672 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:06.672 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:15:06.672 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T11:15:06.672 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:06.672 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:15:06.938 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados-devel 2024-09-18T11:15:07.529 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:28 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:15:07.668 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:15:07.669 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:07.669 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-09-18T11:15:07.669 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:07.669 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:15:07.669 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 126 k 2024-09-18T11:15:07.669 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:07.669 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:15:07.669 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:07.669 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-18T11:15:07.669 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:07.669 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 126 k 2024-09-18T11:15:07.669 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 448 k 2024-09-18T11:15:07.670 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:15:07.870 INFO:teuthology.orchestra.run.smithi157.stdout:librados-devel-17.2.7-1645.g7061ea6c.el9.x86_64 628 kB/s | 126 kB 00:00 2024-09-18T11:15:07.871 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:15:07.871 INFO:teuthology.orchestra.run.smithi157.stdout:Total 623 kB/s | 126 kB 00:00 2024-09-18T11:15:07.871 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:15:07.877 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:15:07.877 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:15:07.941 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:15:07.941 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:15:08.097 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:15:08.192 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:15:08.626 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:15:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:15:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:08.903 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:15:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:15:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:08.903 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:15:09.154 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs2 2024-09-18T11:15:09.741 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:30 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:15:09.832 INFO:teuthology.orchestra.run.smithi157.stdout:Package libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:15:09.875 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:15:09.876 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-18T11:15:09.876 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:15:09.946 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs-devel 2024-09-18T11:15:10.532 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:31 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:15:10.664 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:15:10.665 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:10.665 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-09-18T11:15:10.665 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:10.665 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:15:10.665 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 25 k 2024-09-18T11:15:10.665 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:10.665 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:15:10.665 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:10.665 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-18T11:15:10.665 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:10.666 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 25 k 2024-09-18T11:15:10.666 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 110 k 2024-09-18T11:15:10.666 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:15:10.807 INFO:teuthology.orchestra.run.smithi157.stdout:libcephfs-devel-17.2.7-1645.g7061ea6c.el9.x86_6 177 kB/s | 25 kB 00:00 2024-09-18T11:15:10.808 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:15:10.808 INFO:teuthology.orchestra.run.smithi157.stdout:Total 175 kB/s | 25 kB 00:00 2024-09-18T11:15:10.808 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:15:10.812 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:15:10.812 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:15:10.814 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:15:10.814 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:15:10.898 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:15:11.021 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:15:11.370 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:15:11.604 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:15:11.604 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:11.604 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:15:11.604 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:15:11.605 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:11.605 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:15:11.779 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados2 2024-09-18T11:15:12.356 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:33 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:15:12.444 INFO:teuthology.orchestra.run.smithi157.stdout:Package librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:15:12.487 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:15:12.487 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-18T11:15:12.488 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:15:12.561 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librbd1 2024-09-18T11:15:13.132 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:34 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:15:13.221 INFO:teuthology.orchestra.run.smithi157.stdout:Package librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:15:13.264 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:15:13.265 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-18T11:15:13.265 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:15:13.345 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rados 2024-09-18T11:15:13.927 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:34 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:15:14.017 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:15:14.061 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:15:14.062 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-18T11:15:14.062 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:15:14.139 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rgw 2024-09-18T11:15:14.720 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:35 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:15:14.810 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:15:14.852 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:15:14.853 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-18T11:15:14.853 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:15:14.931 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-cephfs 2024-09-18T11:15:15.509 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:36 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:15:15.599 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:15:15.642 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:15:15.643 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-18T11:15:15.643 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:15:15.711 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rbd 2024-09-18T11:15:16.291 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:37 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:15:16.382 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T11:15:16.426 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:15:16.426 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-18T11:15:16.427 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:15:16.495 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-fuse 2024-09-18T11:15:17.067 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:38 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:15:17.198 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:15:17.198 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:17.199 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-09-18T11:15:17.199 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:17.199 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:15:17.199 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 82 k 2024-09-18T11:15:17.199 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:17.199 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:15:17.199 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:17.199 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-18T11:15:17.199 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:17.199 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 82 k 2024-09-18T11:15:17.199 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 222 k 2024-09-18T11:15:17.199 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:15:17.394 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-fuse-17.2.7-1645.g7061ea6c.el9.x86_64.rpm 421 kB/s | 82 kB 00:00 2024-09-18T11:15:17.395 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:15:17.395 INFO:teuthology.orchestra.run.smithi157.stdout:Total 418 kB/s | 82 kB 00:00 2024-09-18T11:15:17.395 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:15:17.401 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:15:17.402 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:15:17.464 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:15:17.464 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:15:17.615 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:15:17.715 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:15:18.099 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:15:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:15:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:18.353 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:15:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:15:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:18.353 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:15:18.641 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-mirror 2024-09-18T11:15:19.241 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:40 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:15:19.378 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:15:19.379 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:19.379 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-09-18T11:15:19.379 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:19.379 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:15:19.379 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 2.9 M 2024-09-18T11:15:19.379 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:19.379 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:15:19.379 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:19.379 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-18T11:15:19.380 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:19.380 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 2.9 M 2024-09-18T11:15:19.380 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 12 M 2024-09-18T11:15:19.380 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:15:19.715 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-mirror-17.2.7-1645.g7061ea6c.el9.x86_64.rpm 8.8 MB/s | 2.9 MB 00:00 2024-09-18T11:15:19.715 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:15:19.716 INFO:teuthology.orchestra.run.smithi157.stdout:Total 8.7 MB/s | 2.9 MB 00:00 2024-09-18T11:15:19.716 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:15:19.724 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:15:19.725 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:15:19.806 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:15:19.807 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:15:20.226 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:15:20.290 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:15:20.315 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:15:20.316 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T11:15:20.316 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-18T11:15:20.316 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T11:15:20.316 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T11:15:20.316 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:21.407 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:15:21.407 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:21.407 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:15:21.407 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:15:21.407 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:21.407 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:15:21.643 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-nbd 2024-09-18T11:15:22.230 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:43 ago on Wed 18 Sep 2024 11:13:39 AM UTC. 2024-09-18T11:15:22.370 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-18T11:15:22.371 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:22.371 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-09-18T11:15:22.371 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:22.371 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-18T11:15:22.371 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 165 k 2024-09-18T11:15:22.372 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:22.372 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-18T11:15:22.372 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-18T11:15:22.372 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-18T11:15:22.372 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:22.372 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 165 k 2024-09-18T11:15:22.372 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 477 k 2024-09-18T11:15:22.372 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-18T11:15:22.585 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-nbd-17.2.7-1645.g7061ea6c.el9.x86_64.rpm 774 kB/s | 165 kB 00:00 2024-09-18T11:15:22.586 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-18T11:15:22.586 INFO:teuthology.orchestra.run.smithi157.stdout:Total 769 kB/s | 165 kB 00:00 2024-09-18T11:15:22.587 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-18T11:15:22.595 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-18T11:15:22.595 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-18T11:15:22.662 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-18T11:15:22.662 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-18T11:15:22.817 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-18T11:15:22.917 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:15:23.354 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:15:23.602 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T11:15:23.603 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:23.603 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-18T11:15:23.603 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T11:15:23.603 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-18T11:15:23.603 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-18T11:15:23.874 DEBUG:teuthology.parallel:result is None 2024-09-18T11:15:23.875 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:15:24.019 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-18T11:15:24.041 INFO:teuthology.orchestra.run.smithi157.stdout:17.2.7-1645.g7061ea6c.el9 2024-09-18T11:15:24.042 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1645.g7061ea6c.el9 2024-09-18T11:15:24.042 INFO:teuthology.task.install:The correct ceph version 17.2.7-1645.g7061ea6c is installed. 2024-09-18T11:15:24.044 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-18T11:15:24.044 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-18T11:15:24.044 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T11:15:24.109 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-18T11:15:24.109 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-18T11:15:24.109 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T11:15:24.173 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T11:15:24.238 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-18T11:15:24.238 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-18T11:15:24.238 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T11:15:24.303 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T11:15:24.368 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-18T11:15:24.368 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-18T11:15:24.368 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T11:15:24.435 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T11:15:24.499 INFO:teuthology.run_tasks:Running task workunit... 2024-09-18T11:15:24.513 INFO:tasks.workunit:Pulling workunits from ref 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:15:24.514 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-18T11:15:24.514 INFO:tasks.workunit:timeout=3h 2024-09-18T11:15:24.514 INFO:tasks.workunit:cleanup=True 2024-09-18T11:15:24.514 DEBUG:teuthology.orchestra.run.smithi157:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-18T11:15:24.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:15:24.555 INFO:teuthology.orchestra.run.smithi157.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-18T11:15:24.555 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-18T11:15:24.610 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-18T11:15:24.611 DEBUG:teuthology.orchestra.run.smithi157:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-18T11:15:24.667 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:15:24.724 INFO:tasks.workunit.client.0.smithi157.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-18T11:16:15.581 INFO:tasks.workunit.client.0.smithi157.stderr:Note: switching to '7061ea6ca50908d104e9f3979dc615e118d9e153'. 2024-09-18T11:16:15.581 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-18T11:16:15.581 INFO:tasks.workunit.client.0.smithi157.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-18T11:16:15.582 INFO:tasks.workunit.client.0.smithi157.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-18T11:16:15.582 INFO:tasks.workunit.client.0.smithi157.stderr:state without impacting any branches by switching back to a branch. 2024-09-18T11:16:15.582 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-18T11:16:15.582 INFO:tasks.workunit.client.0.smithi157.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-18T11:16:15.582 INFO:tasks.workunit.client.0.smithi157.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-18T11:16:15.582 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-18T11:16:15.582 INFO:tasks.workunit.client.0.smithi157.stderr: git switch -c 2024-09-18T11:16:15.582 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-18T11:16:15.582 INFO:tasks.workunit.client.0.smithi157.stderr:Or undo this operation with: 2024-09-18T11:16:15.582 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-18T11:16:15.582 INFO:tasks.workunit.client.0.smithi157.stderr: git switch - 2024-09-18T11:16:15.582 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-18T11:16:15.582 INFO:tasks.workunit.client.0.smithi157.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-18T11:16:15.583 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-18T11:16:15.583 INFO:tasks.workunit.client.0.smithi157.stderr:HEAD is now at 7061ea6ca50 Merge branch 'wip-65946-quincy' of https://github.com/amathuria/ceph into wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T11:16:15.585 DEBUG:teuthology.orchestra.run.smithi157:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-18T11:16:15.642 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-18T11:16:15.642 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-18T11:16:15.697 INFO:tasks.workunit:Running workunits matching misc on client.0... 2024-09-18T11:16:15.698 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2024-09-18T11:16:15.698 DEBUG:teuthology.orchestra.run.smithi157:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2024-09-18T11:16:15.769 INFO:tasks.workunit.client.0.smithi157.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T11:16:15.775 INFO:tasks.workunit.client.0.smithi157.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T11:16:15.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T11:16:15.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T11:16:15.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T11:16:15.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T11:16:15.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T11:16:15.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T11:16:15.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T11:16:15.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T11:16:15.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/mclock-config 2024-09-18T11:16:15.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2024-09-18T11:16:15.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2024-09-18T11:16:15.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-09-18T11:16:15.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-09-18T11:16:15.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2024-09-18T11:16:15.777 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2024-09-18T11:16:15.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none ' 2024-09-18T11:16:15.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-09-18T11:16:15.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2024-09-18T11:16:15.780 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2024-09-18T11:16:15.780 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T11:16:15.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2024-09-18T11:16:15.782 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_profile_builtin_to_custom 2024-09-18T11:16:15.782 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_profile_custom_to_builtin 2024-09-18T11:16:15.782 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_profile_disallow_builtin_params_modify 2024-09-18T11:16:15.782 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_profile_disallow_builtin_params_override 2024-09-18T11:16:15.782 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_recovery_limit_adjustment_mclock' 2024-09-18T11:16:15.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-18T11:16:15.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-18T11:16:15.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-18T11:16:15.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-18T11:16:15.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:16:15.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:16:15.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:16:15.783 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:16:15.783 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:16:15.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:16:15.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:16:15.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:16:15.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:16:15.784 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:16:15.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:16:15.784 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:16:15.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:16:15.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:16:15.785 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:16:15.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:16:15.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:16:15.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:16:15.787 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:16:15.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:16:15.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:16:15.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:16:15.789 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:16:15.789 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:15.789 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:16:15.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:16:15.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:16:15.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:16:15.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-18T11:16:15.791 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:16:15.791 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:15.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:16:15.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26992 2024-09-18T11:16:15.794 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:16:15.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T11:16:15.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T11:16:15.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:16:15.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-18T11:16:15.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2024-09-18T11:16:15.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2024-09-18T11:16:15.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2024-09-18T11:16:15.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-18T11:16:15.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-18T11:16:15.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:16:15.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:16:15.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:16:15.795 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:16:15.795 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:16:15.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:16:15.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:16:15.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:16:15.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:16:15.796 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:16:15.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:16:15.796 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:16:15.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:16:15.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:16:15.797 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:16:15.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:16:15.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:16:15.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:16:15.798 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:16:15.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:16:15.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:16:15.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:16:15.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:16:15.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:15.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:16:15.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:16:15.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:16:15.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:16:15.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-18T11:16:15.803 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:16:15.803 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:15.804 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:16:15.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26992 2024-09-18T11:16:15.805 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:16:15.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:16:15.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:16:15.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-18T11:16:15.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2024-09-18T11:16:15.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-18T11:16:15.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:16:15.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:16:15.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:16:15.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-18T11:16:15.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-18T11:16:16.188 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:16:16.188 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:16:16.188 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:16:16.188 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:16:16.188 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:16.188 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:16:16.188 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:16:16.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:16:16.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:16:16.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:16:16.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:16:16.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:16:16.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:16:16.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:16:16.432 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:16:16.433 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:16:16.433 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:16:16.433 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:16:16.433 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:16.433 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:16:16.433 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-mon.a.asok 2024-09-18T11:16:16.433 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:16:16.434 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26992/ceph-mon.a.asok config get fsid 2024-09-18T11:16:16.575 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:16:16.575 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:16:16.575 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:16:16.575 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:16:16.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:16:16.576 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:16:16.576 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:16:16.576 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:16:16.576 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:16:16.576 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:16.576 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:16:16.576 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-mon.a.asok 2024-09-18T11:16:16.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:16:16.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26992/ceph-mon.a.asok config get mon_host 2024-09-18T11:16:16.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-09-18T11:16:16.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-18T11:16:16.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:16:16.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:16:16.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:16:16.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-18T11:16:16.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:16:17.486 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:16:17.486 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:16:17.486 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:16:17.486 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:16:17.487 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:17.487 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:16:17.487 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:16:17.487 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:16:17.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:16:17.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-18T11:16:17.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-18T11:16:17.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:16:17.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:16:17.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:16:17.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-18T11:16:17.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-18T11:16:17.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:16:17.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:16:17.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:16:17.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-18T11:16:17.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-18T11:16:17.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:16:17.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:16:17.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-18T11:16:17.514 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:16:17.514 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:16:17.514 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:16:17.515 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:16:17.515 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:17.515 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:16:17.515 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:16:17.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:16:17.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:16:17.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:16:17.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:16:17.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-18T11:16:17.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-18T11:16:17.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:16:17.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:16:17.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:16:17.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:16:17.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:16:17.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-18T11:16:17.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-18T11:16:17.516 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:16:17.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=77bf2a8b-8c9b-4edb-a0c9-70a3487c6e4e 2024-09-18T11:16:17.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 77bf2a8b-8c9b-4edb-a0c9-70a3487c6e4e' 2024-09-18T11:16:17.516 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 77bf2a8b-8c9b-4edb-a0c9-70a3487c6e4e 2024-09-18T11:16:17.517 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:16:17.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCBtupmaJRQHxAAnMBY+3mXkATT8lxr0r3yNQ== 2024-09-18T11:16:17.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCBtupmaJRQHxAAnMBY+3mXkATT8lxr0r3yNQ=="}' 2024-09-18T11:16:17.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 77bf2a8b-8c9b-4edb-a0c9-70a3487c6e4e -i td/mclock-config/0/new.json 2024-09-18T11:16:17.839 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:16:17.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-18T11:16:17.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCBtupmaJRQHxAAnMBY+3mXkATT8lxr0r3yNQ== --osd-uuid 77bf2a8b-8c9b-4edb-a0c9-70a3487c6e4e 2024-09-18T11:16:17.905 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:16:17.904+0000 7fd08c520440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:16:17.907 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:16:17.906+0000 7fd08c520440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:16:17.908 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:16:17.907+0000 7fd08c520440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:16:17.909 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:16:17.907+0000 7fd08c520440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:16:17.909 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:16:17.907+0000 7fd08c520440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:16:17.909 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:16:17.907+0000 7fd08c520440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:16:17.951 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:16:17.950+0000 7fd08c520440 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-18T11:16:24.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-18T11:16:24.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:16:24.198 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:16:24.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:16:24.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:16:24.711 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:16:24.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:16:24.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-18T11:16:24.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:16:24.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:16:24.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:16:24.732 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:16:24.731+0000 7faf7b411440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:16:24.734 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:16:24.733+0000 7faf7b411440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:16:24.735 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:16:24.734+0000 7faf7b411440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:16:25.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:16:25.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:16:25.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:16:25.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:16:25.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:16:25.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:25.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:16:25.063 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:16:25.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:25.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:25.253 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:16:25.252+0000 7faf7b411440 -1 Falling back to public interface 2024-09-18T11:16:25.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:26.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:26.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:26.417 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:16:26.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:16:26.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:26.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:26.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:27.735 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:16:27.734+0000 7faf7b411440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:16:27.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:27.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:27.815 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:16:27.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:16:27.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:27.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:28.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:29.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:29.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:29.330 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:16:29.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:16:29.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:29.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:29.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:30.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:30.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:30.717 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:16:30.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:16:30.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:30.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:31.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:32.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:32.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:32.083 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:16:32.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:16:32.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:32.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:32.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:33.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:33.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:33.600 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:16:33.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:16:33.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:33.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:33.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:34.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:34.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:34.994 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:16:34.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:16:34.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:34.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:35.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:36.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:36.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:36.426 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:16:36.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:16:36.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:36.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:36.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:37.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:37.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:37.809 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:16:37.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:16:37.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:37.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:38.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:39.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:39.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:39.186 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:16:39.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:16:39.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:39.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:40.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:41.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:41.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:41.006 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:16:41.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:16:41.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:41.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:41.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:42.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:42.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:42.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:16:42.644 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:16:42.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:42.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:43.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:44.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:44.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:44.689 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:16:44.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:16:44.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:44.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:45.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:46.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:46.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:46.574 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:16:46.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:16:46.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:46.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:47.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:48.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:48.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:48.594 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:16:48.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:16:48.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:48.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:49.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:50.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:50.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:50.083 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:16:50.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:16:50.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:50.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:50.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:51.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:51.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:51.800 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:16:51.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:16:51.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:51.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:52.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:53.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:53.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:53.197 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T11:16:53.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:16:53.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:53.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:16:53.549 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/28101,v1:127.0.0.1:6803/28101] [v2:127.0.0.1:6804/28101,v1:127.0.0.1:6805/28101] exists,up 77bf2a8b-8c9b-4edb-a0c9-70a3487c6e4e 2024-09-18T11:16:53.551 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:16:53.551 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:16:53.551 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:16:53.551 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-18T11:16:53.551 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2024-09-18T11:16:53.551 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-18T11:16:53.551 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:16:53.552 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:16:53.552 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:16:53.552 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:53.552 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:16:53.552 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:16:53.553 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-18T11:16:53.553 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_max_backfills 2024-09-18T11:16:53.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2024-09-18T11:16:53.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2024-09-18T11:16:53.631 INFO:tasks.workunit.client.0.smithi157.stdout:osd_max_backfills: 1 2024-09-18T11:16:53.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-09-18T11:16:54.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-09-18T11:16:56.128 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-18T11:16:56.129 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2024-09-18T11:16:56.129 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-18T11:16:56.129 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:16:56.129 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:16:56.130 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:16:56.130 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:56.130 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:16:56.130 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:16:56.130 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-18T11:16:56.130 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_max_backfills 2024-09-18T11:16:56.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2024-09-18T11:16:56.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-18T11:16:56.220 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-18T11:16:56.221 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2024-09-18T11:16:56.221 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-18T11:16:56.221 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:16:56.221 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:16:56.221 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:16:56.221 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:56.221 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:16:56.221 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:16:56.221 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-18T11:16:56.221 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok dump_recovery_reservations 2024-09-18T11:16:56.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-09-18T11:16:56.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-18T11:16:56.304 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-18T11:16:56.305 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2024-09-18T11:16:56.305 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-18T11:16:56.305 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:16:56.305 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:16:56.305 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:16:56.305 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:56.305 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:16:56.305 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:16:56.305 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-18T11:16:56.305 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok dump_recovery_reservations 2024-09-18T11:16:56.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-09-18T11:16:56.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-18T11:16:56.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-09-18T11:16:56.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-09-18T11:16:57.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-09-18T11:16:59.273 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-18T11:16:59.274 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2024-09-18T11:16:59.274 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-18T11:16:59.274 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:16:59.274 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:16:59.274 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:16:59.274 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:59.274 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:16:59.274 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:16:59.274 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-18T11:16:59.274 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_max_backfills 2024-09-18T11:16:59.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-18T11:16:59.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-18T11:16:59.362 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-18T11:16:59.362 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2024-09-18T11:16:59.363 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-18T11:16:59.364 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:16:59.364 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:16:59.364 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:16:59.364 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:59.364 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:16:59.364 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:16:59.364 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-18T11:16:59.364 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok dump_recovery_reservations 2024-09-18T11:16:59.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-18T11:16:59.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-18T11:16:59.447 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-18T11:16:59.447 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2024-09-18T11:16:59.447 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-18T11:16:59.447 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:16:59.447 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:16:59.447 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:16:59.447 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:59.448 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:16:59.448 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:16:59.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-18T11:16:59.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok dump_recovery_reservations 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:16:59.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:16:59.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2024-09-18T11:17:00.248 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already down. 2024-09-18T11:17:00.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2024-09-18T11:17:00.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-09-18T11:17:00.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:17:00.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:17:00.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:17:00.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:00.259 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:17:00.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:17:00.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:00.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-09-18T11:17:00.674 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/28101,v1:127.0.0.1:6803/28101] [v2:127.0.0.1:6804/28101,v1:127.0.0.1:6805/28101] exists 77bf2a8b-8c9b-4edb-a0c9-70a3487c6e4e 2024-09-18T11:17:00.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:17:00.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:17:00.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:17:00.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2024-09-18T11:17:00.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/mclock-config 2024-09-18T11:17:00.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:17:00.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T11:17:00.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:17:00.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/mclock-config/0 2024-09-18T11:17:00.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-18T11:17:00.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:17:00.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:17:00.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:17:00.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-18T11:17:00.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-18T11:17:00.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:17:00.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:17:00.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-18T11:17:00.679 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:17:00.680 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:17:00.680 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:17:00.680 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:17:00.680 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:00.680 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:00.680 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:17:00.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:17:00.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:17:00.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-18T11:17:00.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-18T11:17:00.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:17:00.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:17:00.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:17:00.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:17:00.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:17:00.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2024-09-18T11:17:00.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/mclock-config/0 2024-09-18T11:17:00.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T11:17:00.681 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:17:00.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2024-09-18T11:17:00.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/mclock-config/0/whoami 2024-09-18T11:17:00.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T11:17:00.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:17:00.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:17:00.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:17:00.698 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:00.697+0000 7f2ab931d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:00.723 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:00.722+0000 7f2ab931d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:00.724 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:00.723+0000 7f2ab931d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:01.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T11:17:01.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:17:01.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:17:01.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:17:01.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:17:01.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:01.138 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:17:01.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:17:01.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:01.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:01.238 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:01.237+0000 7f2ab931d440 -1 Falling back to public interface 2024-09-18T11:17:01.531 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:02.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:02.533 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:17:02.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:02.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:17:02.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:02.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:02.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:03.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:03.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:03.969 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:17:03.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:17:03.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:03.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:04.086 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:04.084+0000 7f2ab931d440 -1 osd.0 6 log_to_monitors true 2024-09-18T11:17:04.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:05.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:05.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:05.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:17:05.353 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:17:05.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:05.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:05.706 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/29270,v1:127.0.0.1:6803/29270] [v2:127.0.0.1:6804/29270,v1:127.0.0.1:6805/29270] exists,up 77bf2a8b-8c9b-4edb-a0c9-70a3487c6e4e 2024-09-18T11:17:05.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:17:05.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:17:05.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:17:05.708 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-18T11:17:05.708 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2024-09-18T11:17:05.709 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-18T11:17:05.709 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:17:05.709 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:17:05.709 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:17:05.709 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:05.709 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:05.709 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:17:05.709 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-18T11:17:05.709 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_max_backfills 2024-09-18T11:17:05.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-18T11:17:05.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-18T11:17:05.787 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-18T11:17:05.787 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2024-09-18T11:17:05.787 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-18T11:17:05.788 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:17:05.788 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:17:05.788 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:17:05.788 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:05.788 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:05.788 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:17:05.788 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-18T11:17:05.788 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok dump_recovery_reservations 2024-09-18T11:17:05.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-18T11:17:05.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-18T11:17:05.869 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-18T11:17:05.869 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2024-09-18T11:17:05.869 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-18T11:17:05.869 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:17:05.869 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:17:05.869 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:17:05.869 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:05.869 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:05.870 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:17:05.870 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-18T11:17:05.870 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok dump_recovery_reservations 2024-09-18T11:17:05.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-18T11:17:05.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-18T11:17:05.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2024-09-18T11:17:05.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:17:05.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:17:05.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:17:05.951 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:17:05.951 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:17:05.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:17:05.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:17:05.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:17:06.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:17:06.057 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:17:06.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:17:06.058 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:17:06.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:17:06.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:17:06.059 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:17:06.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:06.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:17:06.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:17:06.060 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:06.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:17:06.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:17:06.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:17:06.068 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:17:06.068 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:06.068 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:06.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:17:06.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:17:06.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:17:06.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-18T11:17:06.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:17:06.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:17:06.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:17:06.070 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:17:06.070 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:17:06.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:17:06.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:17:06.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:17:06.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:17:06.071 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:17:06.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:17:06.072 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:17:06.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:17:06.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:17:06.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:17:06.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:06.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:17:06.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:17:06.074 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:06.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:17:06.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:17:06.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:17:06.077 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:17:06.077 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:06.077 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:06.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:17:06.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:17:06.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:17:06.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-18T11:17:06.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-18T11:17:06.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-18T11:17:06.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-18T11:17:06.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:17:06.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:17:06.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:17:06.080 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:17:06.080 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:17:06.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:17:06.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:17:06.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:17:06.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:17:06.081 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:17:06.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:17:06.081 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:17:06.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:17:06.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:17:06.082 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:17:06.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:06.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:17:06.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:17:06.084 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:06.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:17:06.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:17:06.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:17:06.086 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:17:06.086 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:06.086 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:06.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:17:06.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:17:06.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:17:06.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-18T11:17:06.088 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:17:06.088 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:06.089 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:06.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26992 2024-09-18T11:17:06.090 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:17:06.090 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:17:06.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:17:06.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-18T11:17:06.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2024-09-18T11:17:06.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2024-09-18T11:17:06.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2024-09-18T11:17:06.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-18T11:17:06.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:17:06.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:17:06.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:17:06.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-18T11:17:06.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-18T11:17:06.434 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:17:06.434 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:17:06.434 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:17:06.435 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:17:06.435 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:06.435 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:06.435 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:17:06.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:17:06.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:17:06.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:17:06.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:17:06.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:17:06.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:17:06.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:17:06.686 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:17:06.686 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:17:06.687 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:17:06.687 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:17:06.687 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:06.687 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:06.688 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-mon.a.asok 2024-09-18T11:17:06.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:17:06.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26992/ceph-mon.a.asok config get fsid 2024-09-18T11:17:06.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:17:06.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:17:06.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:17:06.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:17:06.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:17:06.764 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:17:06.765 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:17:06.765 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:17:06.765 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:17:06.765 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:06.765 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:06.765 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-mon.a.asok 2024-09-18T11:17:06.765 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:17:06.765 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26992/ceph-mon.a.asok config get mon_host 2024-09-18T11:17:06.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2024-09-18T11:17:06.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-18T11:17:06.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:17:06.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:17:06.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:17:06.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-18T11:17:06.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:17:07.672 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:17:07.672 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:17:07.672 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:17:07.672 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:17:07.672 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:07.673 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:07.673 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:17:07.673 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:17:07.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:17:07.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-18T11:17:07.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-18T11:17:07.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:17:07.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:17:07.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:17:07.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-18T11:17:07.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-18T11:17:07.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:17:07.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:17:07.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:17:07.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-18T11:17:07.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-18T11:17:07.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:17:07.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:17:07.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-18T11:17:07.703 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:17:07.703 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:17:07.703 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:17:07.704 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:17:07.704 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:07.704 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:07.704 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:17:07.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:17:07.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:17:07.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:17:07.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:17:07.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-18T11:17:07.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-18T11:17:07.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:17:07.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:17:07.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:17:07.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:17:07.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:17:07.706 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-18T11:17:07.706 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-18T11:17:07.706 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:17:07.706 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=26e3a025-e206-4f25-b65a-a7ecdd539050 2024-09-18T11:17:07.706 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 26e3a025-e206-4f25-b65a-a7ecdd539050 2024-09-18T11:17:07.707 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 26e3a025-e206-4f25-b65a-a7ecdd539050' 2024-09-18T11:17:07.707 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:17:07.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCztupmPx9MKhAALK0mdU8G+8fGWeR1ex40nQ== 2024-09-18T11:17:07.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCztupmPx9MKhAALK0mdU8G+8fGWeR1ex40nQ=="}' 2024-09-18T11:17:07.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 26e3a025-e206-4f25-b65a-a7ecdd539050 -i td/mclock-config/0/new.json 2024-09-18T11:17:08.007 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:17:08.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-18T11:17:08.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCztupmPx9MKhAALK0mdU8G+8fGWeR1ex40nQ== --osd-uuid 26e3a025-e206-4f25-b65a-a7ecdd539050 2024-09-18T11:17:08.040 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:08.038+0000 7f5707609440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:08.053 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:08.051+0000 7f5707609440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:08.054 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:08.052+0000 7f5707609440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:08.054 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:08.053+0000 7f5707609440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:17:08.054 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:08.053+0000 7f5707609440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:17:08.054 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:08.053+0000 7f5707609440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:17:08.085 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:08.083+0000 7f5707609440 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-18T11:17:13.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-18T11:17:13.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:17:13.460 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:17:13.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:17:13.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:17:13.944 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:17:13.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:17:13.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-18T11:17:13.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:17:13.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:17:13.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:17:13.967 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:13.965+0000 7f4f99b25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:13.968 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:13.967+0000 7f4f99b25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:13.969 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:13.968+0000 7f4f99b25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:14.293 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:17:14.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:17:14.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:17:14.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:17:14.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:17:14.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:14.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:17:14.294 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:17:14.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:14.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:14.499 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:14.498+0000 7f4f99b25440 -1 Falling back to public interface 2024-09-18T11:17:14.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:15.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:15.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:15.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:17:15.658 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:17:15.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:15.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:16.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:16.970 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:16.968+0000 7f4f99b25440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:17:17.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:17.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:17.010 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:17:17.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:17:17.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:17.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:17.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:18.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:18.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:18.515 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:17:18.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:17:18.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:18.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:18.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:19.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:19.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:19.897 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:17:19.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:17:19.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:19.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:20.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:21.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:21.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:21.315 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:17:21.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:17:21.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:21.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:21.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:22.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:22.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:22.737 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:17:22.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:17:22.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:22.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:23.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:24.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:24.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:24.126 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:17:24.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:17:24.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:24.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:24.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:25.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:25.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:25.644 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:17:25.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:17:25.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:25.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:26.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:27.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:27.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:27.022 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:17:27.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:17:27.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:27.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:27.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:28.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:28.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:28.422 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:17:28.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:17:28.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:28.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:29.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:30.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:30.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:30.306 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:17:30.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:17:30.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:30.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:30.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:31.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:31.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:31.855 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:17:31.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:17:31.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:31.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:32.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:33.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:33.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:33.684 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:17:33.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:17:33.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:33.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:34.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:35.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:35.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:35.311 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:17:35.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:17:35.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:35.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:35.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:36.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:36.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:36.784 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:17:36.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:17:36.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:36.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:37.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:38.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:38.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:38.510 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:17:38.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:17:38.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:38.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:39.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:40.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:40.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:40.047 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:17:40.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:17:40.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:40.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:40.425 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/30972,v1:127.0.0.1:6803/30972] [v2:127.0.0.1:6804/30972,v1:127.0.0.1:6805/30972] exists,up 26e3a025-e206-4f25-b65a-a7ecdd539050 2024-09-18T11:17:40.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:17:40.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:17:40.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:17:40.426 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2024-09-18T11:17:40.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2024-09-18T11:17:40.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2024-09-18T11:17:40.775 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-09-18T11:17:40.775 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-18T11:17:40.775 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:17:40.775 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:17:40.776 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:17:40.776 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:40.776 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:40.776 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:17:40.776 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-18T11:17:40.776 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_profile 2024-09-18T11:17:40.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2024-09-18T11:17:40.855 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2024-09-18T11:17:40.856 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2024-09-18T11:17:40.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2024-09-18T11:17:40.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2024-09-18T11:17:40.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2024-09-18T11:17:40.952 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:17:40.953 INFO:tasks.workunit.client.0.smithi157.stdout: "success": "" 2024-09-18T11:17:40.953 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:17:40.962 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-09-18T11:17:40.962 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-18T11:17:40.963 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:17:40.963 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:17:40.963 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:17:40.963 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:40.963 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:40.963 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:17:40.963 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-18T11:17:40.963 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_profile 2024-09-18T11:17:41.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2024-09-18T11:17:41.042 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2024-09-18T11:17:41.042 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2024-09-18T11:17:41.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2024-09-18T11:17:41.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2024-09-18T11:17:41.043 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-09-18T11:17:41.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2024-09-18T11:17:41.044 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-18T11:17:41.044 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:17:41.044 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:17:41.044 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:17:41.044 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:41.044 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:41.044 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:17:41.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-18T11:17:41.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-18T11:17:41.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.300000 2024-09-18T11:17:41.123 INFO:tasks.workunit.client.0.smithi157.stdout:client_res = 0.300000 2024-09-18T11:17:41.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.300000' 2024-09-18T11:17:41.123 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.300000 + 0.1' 2024-09-18T11:17:41.124 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2024-09-18T11:17:41.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.400000 2024-09-18T11:17:41.124 INFO:tasks.workunit.client.0.smithi157.stdout:client_res_new = .400000 2024-09-18T11:17:41.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .400000' 2024-09-18T11:17:41.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-09-18T11:17:41.558 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-18T11:17:41.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.400000 2024-09-18T11:17:41.958 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-09-18T11:17:41.958 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2024-09-18T11:17:41.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2024-09-18T11:17:41.961 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-09-18T11:17:41.961 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2024-09-18T11:17:41.961 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-18T11:17:41.961 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:17:41.961 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:17:41.961 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:17:41.961 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:41.961 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:41.961 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:17:41.962 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-18T11:17:41.962 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-18T11:17:42.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.400000 2024-09-18T11:17:42.040 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-09-18T11:17:42.040 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2024-09-18T11:17:42.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2024-09-18T11:17:42.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2024-09-18T11:17:42.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:17:42.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:17:42.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:17:42.042 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:17:42.042 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:17:42.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:17:42.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:17:42.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:17:42.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:17:42.148 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:17:42.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:17:42.149 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:17:42.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:17:42.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:17:42.150 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:17:42.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:42.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:17:42.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:17:42.151 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:42.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:17:42.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:17:42.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:17:42.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:17:42.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:42.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:42.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:17:42.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:17:42.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:17:42.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-18T11:17:42.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:17:42.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:17:42.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:17:42.160 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:17:42.160 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:17:42.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:17:42.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:17:42.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:17:42.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:17:42.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:17:42.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:17:42.162 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:17:42.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:17:42.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:17:42.164 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:17:42.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:42.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:17:42.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:17:42.165 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:42.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:17:42.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:17:42.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:17:42.168 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:17:42.168 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:42.168 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:42.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:17:42.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:17:42.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:17:42.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-18T11:17:42.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-18T11:17:42.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-18T11:17:42.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-18T11:17:42.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:17:42.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:17:42.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:17:42.170 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:17:42.170 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:17:42.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:17:42.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:17:42.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:17:42.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:17:42.171 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:17:42.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:17:42.172 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:17:42.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:17:42.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:17:42.173 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:17:42.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:42.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:17:42.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:17:42.174 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:42.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:17:42.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:17:42.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:17:42.177 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:17:42.177 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:42.177 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:42.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:17:42.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:17:42.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:17:42.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-18T11:17:42.179 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:17:42.179 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:42.179 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:42.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26992 2024-09-18T11:17:42.182 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:17:42.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:17:42.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:17:42.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-18T11:17:42.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2024-09-18T11:17:42.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2024-09-18T11:17:42.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2024-09-18T11:17:42.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-18T11:17:42.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-18T11:17:42.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:17:42.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:17:42.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:17:42.183 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:17:42.183 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:17:42.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:17:42.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:17:42.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:17:42.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:17:42.184 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:17:42.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:17:42.184 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:17:42.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:17:42.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:17:42.185 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:17:42.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:42.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:17:42.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:17:42.186 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:42.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:17:42.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:17:42.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:17:42.189 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:17:42.189 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:42.189 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:42.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:17:42.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:17:42.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:17:42.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-18T11:17:42.191 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:17:42.191 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:42.191 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:42.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26992 2024-09-18T11:17:42.193 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:17:42.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:17:42.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:17:42.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-18T11:17:42.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2024-09-18T11:17:42.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-18T11:17:42.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:17:42.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:17:42.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:17:42.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-18T11:17:42.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-18T11:17:42.556 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:17:42.557 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:17:42.557 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:17:42.557 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:17:42.557 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:42.557 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:42.557 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:17:42.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:17:42.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:17:42.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:17:42.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:17:42.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:17:42.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:17:42.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:17:42.792 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:17:42.792 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:17:42.793 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:17:42.793 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:17:42.793 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:42.793 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:42.793 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-mon.a.asok 2024-09-18T11:17:42.793 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:17:42.793 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26992/ceph-mon.a.asok config get fsid 2024-09-18T11:17:42.867 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:17:42.867 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:17:42.867 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:17:42.867 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:17:42.867 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:17:42.867 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:17:42.867 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:17:42.867 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:17:42.867 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:17:42.867 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:42.867 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:42.868 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-mon.a.asok 2024-09-18T11:17:42.868 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:17:42.868 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26992/ceph-mon.a.asok config get mon_host 2024-09-18T11:17:42.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2024-09-18T11:17:42.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-18T11:17:42.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:17:42.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:17:42.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:17:42.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-18T11:17:42.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:17:43.770 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:17:43.770 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:17:43.770 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:17:43.770 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:17:43.770 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:43.770 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:43.771 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:17:43.771 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:17:43.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:17:43.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-18T11:17:43.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-18T11:17:43.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:17:43.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:17:43.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:17:43.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-18T11:17:43.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-18T11:17:43.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:17:43.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:17:43.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:17:43.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-18T11:17:43.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-18T11:17:43.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:17:43.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:17:43.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-18T11:17:43.802 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:17:43.802 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:17:43.802 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:17:43.803 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:17:43.803 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:43.803 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:17:43.803 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:17:43.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:17:43.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:17:43.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:17:43.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:17:43.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-18T11:17:43.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-18T11:17:43.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:17:43.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:17:43.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:17:43.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:17:43.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:17:43.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-18T11:17:43.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-18T11:17:43.805 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:17:43.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d57cc576-a068-4f20-8db2-c7b671d64c61 2024-09-18T11:17:43.806 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 d57cc576-a068-4f20-8db2-c7b671d64c61 2024-09-18T11:17:43.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d57cc576-a068-4f20-8db2-c7b671d64c61' 2024-09-18T11:17:43.807 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:17:43.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDXtupmBz0xMBAAq9eHIIOamPjr956vKKWlWg== 2024-09-18T11:17:43.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDXtupmBz0xMBAAq9eHIIOamPjr956vKKWlWg=="}' 2024-09-18T11:17:43.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d57cc576-a068-4f20-8db2-c7b671d64c61 -i td/mclock-config/0/new.json 2024-09-18T11:17:44.113 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:17:44.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-18T11:17:44.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDXtupmBz0xMBAAq9eHIIOamPjr956vKKWlWg== --osd-uuid d57cc576-a068-4f20-8db2-c7b671d64c61 2024-09-18T11:17:44.146 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:44.145+0000 7f1b7a151440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:44.159 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:44.157+0000 7f1b7a151440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:44.160 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:44.158+0000 7f1b7a151440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:44.160 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:44.159+0000 7f1b7a151440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:17:44.160 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:44.159+0000 7f1b7a151440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:17:44.160 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:44.159+0000 7f1b7a151440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:17:44.199 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:44.197+0000 7f1b7a151440 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-18T11:17:49.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-18T11:17:49.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:17:49.526 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:17:49.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:17:49.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:17:50.005 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:17:50.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:17:50.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-18T11:17:50.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:17:50.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:17:50.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:17:50.027 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:50.025+0000 7fcff9e39440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:50.029 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:50.028+0000 7fcff9e39440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:50.030 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:50.028+0000 7fcff9e39440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:50.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:17:50.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:17:50.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:17:50.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:17:50.351 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:17:50.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:17:50.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:50.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:17:50.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:50.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:50.552 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:50.551+0000 7fcff9e39440 -1 Falling back to public interface 2024-09-18T11:17:50.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:51.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:51.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:51.710 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:17:51.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:17:51.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:51.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:52.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:53.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:53.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:53.054 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:17:53.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:17:53.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:53.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:53.098 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:17:53.096+0000 7fcff9e39440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:17:53.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:54.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:54.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:54.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:17:54.467 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:17:54.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:54.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:54.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:55.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:55.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:55.874 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:17:55.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:17:55.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:55.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:56.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:57.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:57.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:57.251 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:17:57.252 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:17:57.252 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:57.252 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:57.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:58.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:58.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:58.606 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:17:58.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:17:58.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:58.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:59.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:00.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:00.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:00.057 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:18:00.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:18:00.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:00.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:00.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:01.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:01.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:01.413 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:18:01.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:18:01.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:01.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:01.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:02.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:02.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:02.906 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:18:02.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:18:02.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:02.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:03.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:04.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:04.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:04.296 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:18:04.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:18:04.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:04.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:04.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:05.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:05.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:05.764 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:18:05.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:18:05.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:05.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:06.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:07.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:07.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:07.201 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:18:07.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:18:07.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:07.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:07.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:08.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:08.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:08.947 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:18:08.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:18:08.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:08.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:09.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:10.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:10.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:10.353 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:18:10.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:18:10.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:10.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:11.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:12.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:12.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:12.057 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:18:12.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:18:12.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:12.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:12.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:13.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:13.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:13.877 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:18:13.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:18:13.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:13.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:14.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:15.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:15.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:15.780 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:18:15.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:18:15.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:15.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:16.192 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/33148,v1:127.0.0.1:6803/33148] [v2:127.0.0.1:6804/33148,v1:127.0.0.1:6805/33148] exists,up d57cc576-a068-4f20-8db2-c7b671d64c61 2024-09-18T11:18:16.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:18:16.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:18:16.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:18:16.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2024-09-18T11:18:16.193 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2024-09-18T11:18:16.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2024-09-18T11:18:16.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2024-09-18T11:18:16.541 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-18T11:18:16.541 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-18T11:18:16.541 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:18:16.541 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:18:16.541 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:18:16.542 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:16.542 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:16.542 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:18:16.542 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-18T11:18:16.542 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_profile 2024-09-18T11:18:16.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2024-09-18T11:18:16.622 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2024-09-18T11:18:16.622 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2024-09-18T11:18:16.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2024-09-18T11:18:16.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-09-18T11:18:16.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2024-09-18T11:18:16.747 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:18:16.748 INFO:tasks.workunit.client.0.smithi157.stdout: "success": "" 2024-09-18T11:18:16.748 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:18:16.758 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-18T11:18:16.758 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-18T11:18:16.758 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:18:16.758 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:18:16.758 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:18:16.758 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:16.758 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:16.758 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:18:16.758 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-18T11:18:16.758 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_profile 2024-09-18T11:18:16.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2024-09-18T11:18:16.837 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2024-09-18T11:18:16.837 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2024-09-18T11:18:16.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2024-09-18T11:18:16.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2024-09-18T11:18:16.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2024-09-18T11:18:16.839 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-18T11:18:16.839 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2024-09-18T11:18:16.839 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-18T11:18:16.839 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:18:16.840 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:18:16.840 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:18:16.840 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:16.840 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:16.840 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:18:16.840 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-18T11:18:16.840 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-18T11:18:16.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.300000 2024-09-18T11:18:16.917 INFO:tasks.workunit.client.0.smithi157.stdout:Original client_res for osd.0 = 0.300000 2024-09-18T11:18:16.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.300000' 2024-09-18T11:18:16.918 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.300000 + 0.1' 2024-09-18T11:18:16.918 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2024-09-18T11:18:16.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.400000 2024-09-18T11:18:16.918 INFO:tasks.workunit.client.0.smithi157.stdout:client_res_new = .400000 2024-09-18T11:18:16.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .400000' 2024-09-18T11:18:16.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .400000 2024-09-18T11:18:17.343 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-18T11:18:17.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.400000 2024-09-18T11:18:17.733 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-18T11:18:17.734 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2024-09-18T11:18:17.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-18T11:18:17.736 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-18T11:18:17.736 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2024-09-18T11:18:17.736 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-18T11:18:17.736 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:18:17.736 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:18:17.736 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:18:17.736 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:17.736 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:17.736 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:18:17.737 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-18T11:18:17.737 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-18T11:18:17.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.400000 2024-09-18T11:18:17.816 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-18T11:18:17.816 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2024-09-18T11:18:17.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-18T11:18:17.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2024-09-18T11:18:17.941 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:18:17.941 INFO:tasks.workunit.client.0.smithi157.stdout: "success": "" 2024-09-18T11:18:17.941 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:18:17.951 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-18T11:18:17.951 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-18T11:18:17.951 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:18:17.951 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:18:17.951 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:18:17.951 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:17.951 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:17.951 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:18:17.951 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-18T11:18:17.951 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_profile 2024-09-18T11:18:18.030 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2024-09-18T11:18:18.030 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2024-09-18T11:18:18.030 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-09-18T11:18:18.030 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-18T11:18:18.375 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.400000 2024-09-18T11:18:18.375 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-18T11:18:18.375 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2024-09-18T11:18:18.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-18T11:18:18.377 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-18T11:18:18.378 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2024-09-18T11:18:18.378 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-18T11:18:18.378 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:18:18.378 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:18:18.378 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:18:18.378 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:18.378 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:18.378 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:18:18.378 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-18T11:18:18.378 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-18T11:18:18.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.400000 2024-09-18T11:18:18.457 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-18T11:18:18.457 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2024-09-18T11:18:18.458 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-18T11:18:18.458 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2024-09-18T11:18:18.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2024-09-18T11:18:23.874 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-18T11:18:24.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2024-09-18T11:18:24.232 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2024-09-18T11:18:24.232 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2024-09-18T11:18:24.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-18T11:18:24.234 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-18T11:18:24.234 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2024-09-18T11:18:24.235 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-18T11:18:24.235 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:18:24.235 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:18:24.235 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:18:24.235 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:24.235 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:24.235 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:18:24.235 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-18T11:18:24.235 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-18T11:18:24.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.300000 2024-09-18T11:18:24.314 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.300000 != 0.300000' 2024-09-18T11:18:24.314 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2024-09-18T11:18:24.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-18T11:18:24.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2024-09-18T11:18:24.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:18:24.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:18:24.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:18:24.316 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:18:24.316 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:18:24.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:18:24.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:18:24.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:18:24.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:18:24.422 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:18:24.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:18:24.423 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:18:24.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:18:24.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:18:24.424 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:18:24.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:18:24.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:18:24.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:18:24.426 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:18:24.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:18:24.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:18:24.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:18:24.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:18:24.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:24.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:24.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:18:24.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:18:24.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:18:24.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-18T11:18:24.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:18:24.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:18:24.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:18:24.434 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:18:24.434 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:18:24.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:18:24.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:18:24.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:18:24.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:18:24.435 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:18:24.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:18:24.436 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:18:24.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:18:24.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:18:24.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:18:24.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:18:24.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:18:24.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:18:24.439 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:18:24.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:18:24.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:18:24.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:18:24.442 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:18:24.442 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:24.442 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:24.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:18:24.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:18:24.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:18:24.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-18T11:18:24.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-18T11:18:24.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-18T11:18:24.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-18T11:18:24.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:18:24.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:18:24.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:18:24.444 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:18:24.444 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:18:24.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:18:24.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:18:24.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:18:24.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:18:24.445 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:18:24.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:18:24.446 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:18:24.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:18:24.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:18:24.447 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:18:24.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:18:24.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:18:24.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:18:24.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:18:24.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:18:24.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:18:24.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:18:24.451 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:18:24.451 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:24.451 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:24.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:18:24.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:18:24.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:18:24.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-18T11:18:24.454 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:18:24.454 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:24.454 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:24.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26992 2024-09-18T11:18:24.456 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:18:24.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:18:24.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:18:24.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-18T11:18:24.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_modify td/mclock-config 2024-09-18T11:18:24.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:305: TEST_profile_disallow_builtin_params_modify: local dir=td/mclock-config 2024-09-18T11:18:24.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:307: TEST_profile_disallow_builtin_params_modify: setup td/mclock-config 2024-09-18T11:18:24.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-18T11:18:24.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-18T11:18:24.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:18:24.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:18:24.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:18:24.457 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:18:24.457 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:18:24.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:18:24.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:18:24.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:18:24.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:18:24.458 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:18:24.458 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:18:24.458 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:18:24.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:18:24.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:18:24.459 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:18:24.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:18:24.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:18:24.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:18:24.461 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:18:24.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:18:24.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:18:24.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:18:24.463 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:18:24.463 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:24.463 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:24.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:18:24.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:18:24.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:18:24.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-18T11:18:24.465 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:18:24.466 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:24.466 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:24.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26992 2024-09-18T11:18:24.467 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:18:24.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:18:24.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:18:24.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-18T11:18:24.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:308: TEST_profile_disallow_builtin_params_modify: run_mon td/mclock-config a 2024-09-18T11:18:24.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-18T11:18:24.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:18:24.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:18:24.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:18:24.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-18T11:18:24.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-18T11:18:24.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:18:24.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:18:24.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:18:24.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:18:24.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:24.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:24.821 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:18:24.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:18:25.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:18:25.031 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:18:25.031 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:18:25.031 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:18:25.031 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:18:25.031 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:18:25.031 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:18:25.031 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:18:25.031 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:18:25.031 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:18:25.031 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:25.031 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:25.031 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-mon.a.asok 2024-09-18T11:18:25.032 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:18:25.032 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26992/ceph-mon.a.asok config get fsid 2024-09-18T11:18:25.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:18:25.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:18:25.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:18:25.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:18:25.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:18:25.106 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:18:25.107 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:18:25.107 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:18:25.107 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:18:25.107 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:25.107 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:25.107 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-mon.a.asok 2024-09-18T11:18:25.107 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:18:25.107 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26992/ceph-mon.a.asok config get mon_host 2024-09-18T11:18:25.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:309: TEST_profile_disallow_builtin_params_modify: run_mgr td/mclock-config x 2024-09-18T11:18:25.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-18T11:18:25.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:18:25.181 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:18:25.181 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:18:25.181 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-18T11:18:25.181 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:18:26.024 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:18:26.024 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:18:26.024 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:18:26.024 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:18:26.024 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:26.024 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:26.024 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:18:26.024 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:18:26.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:18:26.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:311: TEST_profile_disallow_builtin_params_modify: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-18T11:18:26.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-18T11:18:26.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:18:26.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:18:26.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:18:26.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-18T11:18:26.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-18T11:18:26.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:18:26.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:18:26.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:18:26.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-18T11:18:26.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-18T11:18:26.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:18:26.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:18:26.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-18T11:18:26.052 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:18:26.052 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:18:26.052 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:18:26.052 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:18:26.053 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:26.053 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:26.053 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:18:26.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:18:26.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:18:26.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:18:26.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:18:26.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-18T11:18:26.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-18T11:18:26.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:18:26.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:18:26.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:18:26.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:18:26.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:18:26.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-18T11:18:26.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-18T11:18:26.054 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:18:26.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=19439205-870c-4357-9410-bb7409a915bb 2024-09-18T11:18:26.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 19439205-870c-4357-9410-bb7409a915bb' 2024-09-18T11:18:26.055 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 19439205-870c-4357-9410-bb7409a915bb 2024-09-18T11:18:26.055 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:18:26.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACt+pmIjfPAxAA1DWL2LJiOT44cts3GA6usA== 2024-09-18T11:18:26.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACt+pmIjfPAxAA1DWL2LJiOT44cts3GA6usA=="}' 2024-09-18T11:18:26.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 19439205-870c-4357-9410-bb7409a915bb -i td/mclock-config/0/new.json 2024-09-18T11:18:26.368 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:18:26.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-18T11:18:26.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQACt+pmIjfPAxAA1DWL2LJiOT44cts3GA6usA== --osd-uuid 19439205-870c-4357-9410-bb7409a915bb 2024-09-18T11:18:26.402 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:18:26.400+0000 7f6c4e12b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:18:26.413 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:18:26.412+0000 7f6c4e12b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:18:26.414 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:18:26.413+0000 7f6c4e12b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:18:26.415 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:18:26.413+0000 7f6c4e12b440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:18:26.415 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:18:26.413+0000 7f6c4e12b440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:18:26.415 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:18:26.413+0000 7f6c4e12b440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:18:26.454 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:18:26.452+0000 7f6c4e12b440 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-18T11:18:31.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-18T11:18:31.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:18:31.763 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:18:31.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:18:31.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:18:32.250 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:18:32.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:18:32.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-18T11:18:32.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:18:32.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:18:32.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:18:32.271 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:18:32.270+0000 7f43f8e3f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:18:32.273 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:18:32.272+0000 7f43f8e3f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:18:32.274 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:18:32.273+0000 7f43f8e3f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:18:32.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:18:32.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:18:32.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:18:32.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:18:32.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:18:32.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:32.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:18:32.600 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:18:32.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:32.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:32.791 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:18:32.790+0000 7f43f8e3f440 -1 Falling back to public interface 2024-09-18T11:18:32.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:33.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:33.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:33.957 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:18:33.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:18:33.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:33.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:34.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:35.270 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:18:35.268+0000 7f43f8e3f440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:18:35.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:35.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:35.312 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:18:35.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:18:35.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:35.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:35.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:36.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:36.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:36.849 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:18:36.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:18:36.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:36.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:37.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:38.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:38.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:38.235 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:18:38.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:18:38.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:38.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:38.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:39.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:39.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:39.592 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:18:39.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:18:39.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:39.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:40.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:41.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:41.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:41.054 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:18:41.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:18:41.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:41.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:41.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:42.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:42.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:42.418 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:18:42.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:18:42.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:42.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:42.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:43.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:43.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:43.782 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:18:43.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:18:43.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:43.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:44.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:45.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:45.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:45.211 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:18:45.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:18:45.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:45.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:45.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:46.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:46.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:46.573 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:18:46.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:18:46.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:46.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:46.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:47.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:47.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:47.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:18:47.960 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:18:47.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:47.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:48.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:49.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:49.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:49.397 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:18:49.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:18:49.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:49.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:49.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:50.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:50.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:50.826 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:18:50.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:18:50.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:50.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:51.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:52.568 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:52.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:52.569 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:18:52.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:18:52.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:52.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:53.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:54.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:54.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:54.346 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:18:54.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:18:54.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:54.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:55.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:56.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:56.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:56.540 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:18:56.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:18:56.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:56.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:57.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:58.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:58.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:58.204 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:18:58.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:18:58.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:58.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:18:58.558 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/35446,v1:127.0.0.1:6803/35446] [v2:127.0.0.1:6804/35446,v1:127.0.0.1:6805/35446] exists,up 19439205-870c-4357-9410-bb7409a915bb 2024-09-18T11:18:58.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:18:58.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:18:58.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:18:58.559 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_profile 2024-09-18T11:18:58.916 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: local def_mclock_profile=balanced 2024-09-18T11:18:58.916 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:315: TEST_profile_disallow_builtin_params_modify: test balanced = balanced 2024-09-18T11:18:58.917 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_profile 2024-09-18T11:18:58.917 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-18T11:18:58.917 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:18:58.917 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:18:58.917 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:18:58.918 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:58.918 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:58.918 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:18:58.918 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-18T11:18:58.918 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_profile 2024-09-18T11:18:58.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: local 'cur_mclock_profile="high_recovery_ops"' 2024-09-18T11:18:58.998 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: eval echo '"high_recovery_ops"' 2024-09-18T11:18:58.998 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: echo high_recovery_ops 2024-09-18T11:18:58.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: cur_mclock_profile=high_recovery_ops 2024-09-18T11:18:58.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:322: TEST_profile_disallow_builtin_params_modify: test high_recovery_ops = high_recovery_ops 2024-09-18T11:18:58.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-09-18T11:18:58.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: declare -a options 2024-09-18T11:18:58.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:327: TEST_profile_disallow_builtin_params_modify: local retries=10 2024-09-18T11:18:58.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:328: TEST_profile_disallow_builtin_params_modify: local errors=0 2024-09-18T11:18:58.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-09-18T11:18:58.999 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-09-18T11:18:58.999 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-09-18T11:18:58.999 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-18T11:18:58.999 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:18:58.999 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:18:58.999 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:18:58.999 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:58.999 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:18:58.999 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:18:58.999 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-18T11:18:59.000 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-18T11:18:59.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.700000 2024-09-18T11:18:59.077 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.700000 + 0.1' 2024-09-18T11:18:59.077 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-18T11:18:59.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.800000 2024-09-18T11:18:59.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_background_recovery_res .800000 2024-09-18T11:18:59.534 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-09-18T11:18:59.535 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-09-18T11:18:59.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-09-18T11:18:59.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-09-18T11:18:59.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-09-18T11:19:01.556 INFO:tasks.workunit.client.0.smithi157.stdout:Check configuration values - Attempt#: 0 2024-09-18T11:19:01.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-09-18T11:19:01.557 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-09-18T11:19:01.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-09-18T11:19:01.919 INFO:tasks.workunit.client.0.smithi157.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-09-18T11:19:01.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-09-18T11:19:01.919 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .800000' 2024-09-18T11:19:01.920 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-18T11:19:01.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-09-18T11:19:01.921 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-09-18T11:19:01.921 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_background_recovery_res 2024-09-18T11:19:01.921 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-09-18T11:19:01.921 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-09-18T11:19:02.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.700000 2024-09-18T11:19:02.266 INFO:tasks.workunit.client.0.smithi157.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-09-18T11:19:02.267 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-09-18T11:19:02.267 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 == .800000' 2024-09-18T11:19:02.267 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-18T11:19:02.268 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 != 0.700000' 2024-09-18T11:19:02.268 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-18T11:19:02.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-18T11:19:02.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-09-18T11:19:02.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-09-18T11:19:02.270 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-18T11:19:02.271 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:19:02.271 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:19:02.271 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:19:02.271 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:02.271 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:02.271 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:19:02.271 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-18T11:19:02.271 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-18T11:19:02.349 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.700000 2024-09-18T11:19:02.349 INFO:tasks.workunit.client.0.smithi157.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-09-18T11:19:02.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-09-18T11:19:02.350 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 == .800000' 2024-09-18T11:19:02.350 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-18T11:19:02.351 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 != 0.700000' 2024-09-18T11:19:02.351 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-18T11:19:02.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-18T11:19:02.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-09-18T11:19:02.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-09-18T11:19:02.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-09-18T11:19:02.354 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-09-18T11:19:02.354 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-09-18T11:19:02.354 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-18T11:19:02.354 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:19:02.354 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:19:02.354 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:19:02.354 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:02.355 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:02.355 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:19:02.355 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-18T11:19:02.355 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-18T11:19:02.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.300000 2024-09-18T11:19:02.434 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.300000 + 0.1' 2024-09-18T11:19:02.434 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-18T11:19:02.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.400000 2024-09-18T11:19:02.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-09-18T11:19:02.889 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-09-18T11:19:02.889 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-09-18T11:19:02.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-09-18T11:19:02.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-09-18T11:19:02.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-09-18T11:19:04.891 INFO:tasks.workunit.client.0.smithi157.stdout:Check configuration values - Attempt#: 0 2024-09-18T11:19:04.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-09-18T11:19:04.892 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-18T11:19:05.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-09-18T11:19:05.246 INFO:tasks.workunit.client.0.smithi157.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-09-18T11:19:05.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-09-18T11:19:05.248 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2024-09-18T11:19:05.248 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-18T11:19:05.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-09-18T11:19:05.249 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-09-18T11:19:05.249 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2024-09-18T11:19:05.249 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-09-18T11:19:05.250 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-09-18T11:19:05.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.300000 2024-09-18T11:19:05.595 INFO:tasks.workunit.client.0.smithi157.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-09-18T11:19:05.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-09-18T11:19:05.595 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 == .400000' 2024-09-18T11:19:05.595 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-18T11:19:05.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 != 0.300000' 2024-09-18T11:19:05.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-18T11:19:05.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-18T11:19:05.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-09-18T11:19:05.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-09-18T11:19:05.599 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-18T11:19:05.599 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:19:05.599 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:19:05.599 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:19:05.599 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:05.599 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:05.599 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:19:05.599 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-18T11:19:05.600 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-18T11:19:05.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2024-09-18T11:19:05.678 INFO:tasks.workunit.client.0.smithi157.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-09-18T11:19:05.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-09-18T11:19:05.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2024-09-18T11:19:05.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-18T11:19:05.679 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2024-09-18T11:19:05.679 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-18T11:19:05.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-18T11:19:05.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-09-18T11:19:05.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-09-18T11:19:05.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:380: TEST_profile_disallow_builtin_params_modify: teardown td/mclock-config 2024-09-18T11:19:05.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:19:05.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:19:05.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:19:05.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:19:05.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:19:05.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:19:05.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:19:05.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:19:05.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:19:05.787 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:19:05.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:19:05.788 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:19:05.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:19:05.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:19:05.789 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:19:05.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:19:05.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:19:05.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:19:05.791 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:19:05.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:19:05.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:19:05.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:19:05.797 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:19:05.797 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:05.797 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:05.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:19:05.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:19:05.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:19:05.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-18T11:19:05.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:19:05.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:19:05.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:19:05.799 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:19:05.799 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:19:05.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:19:05.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:19:05.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:19:05.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:19:05.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:19:05.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:19:05.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:19:05.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:19:05.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:19:05.803 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:19:05.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:19:05.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:19:05.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:19:05.804 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:19:05.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:19:05.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:19:05.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:19:05.807 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:19:05.807 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:05.807 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:05.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:19:05.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:19:05.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:19:05.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-18T11:19:05.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-18T11:19:05.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-18T11:19:05.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-18T11:19:05.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:19:05.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:19:05.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:19:05.809 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:19:05.810 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:19:05.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:19:05.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:19:05.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:19:05.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:19:05.810 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:19:05.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:19:05.811 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:19:05.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:19:05.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:19:05.812 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:19:05.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:19:05.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:19:05.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:19:05.814 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:19:05.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:19:05.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:19:05.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:19:05.816 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:19:05.816 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:05.816 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:05.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:19:05.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:19:05.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:19:05.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-18T11:19:05.818 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:19:05.818 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:05.818 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:05.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26992 2024-09-18T11:19:05.821 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:19:05.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:19:05.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:19:05.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-18T11:19:05.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_override td/mclock-config 2024-09-18T11:19:05.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:384: TEST_profile_disallow_builtin_params_override: local dir=td/mclock-config 2024-09-18T11:19:05.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:386: TEST_profile_disallow_builtin_params_override: setup td/mclock-config 2024-09-18T11:19:05.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-18T11:19:05.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-18T11:19:05.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:19:05.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:19:05.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:19:05.822 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:19:05.822 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:19:05.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:19:05.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:19:05.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:19:05.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:19:05.823 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:19:05.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:19:05.823 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:19:05.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:19:05.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:19:05.824 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:19:05.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:19:05.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:19:05.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:19:05.826 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:19:05.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:19:05.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:19:05.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:19:05.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:19:05.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:05.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:05.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:19:05.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:19:05.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:19:05.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-18T11:19:05.830 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:19:05.831 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:05.831 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:05.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26992 2024-09-18T11:19:05.832 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:19:05.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:19:05.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:19:05.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-18T11:19:05.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:387: TEST_profile_disallow_builtin_params_override: run_mon td/mclock-config a 2024-09-18T11:19:05.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-18T11:19:05.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:19:05.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:19:05.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:19:05.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-18T11:19:05.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-18T11:19:06.187 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:19:06.187 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:19:06.187 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:19:06.187 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:19:06.187 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:06.187 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:06.187 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:19:06.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:19:06.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:19:06.406 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:19:06.406 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:19:06.406 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:19:06.406 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:19:06.406 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:19:06.406 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:19:06.406 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:19:06.406 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:19:06.406 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:19:06.406 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:06.407 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:06.407 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-mon.a.asok 2024-09-18T11:19:06.407 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:19:06.407 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26992/ceph-mon.a.asok config get fsid 2024-09-18T11:19:06.482 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:19:06.482 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:19:06.483 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:19:06.483 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:19:06.483 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:19:06.483 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:19:06.483 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:19:06.483 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:19:06.483 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:19:06.483 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:06.483 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:06.483 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-mon.a.asok 2024-09-18T11:19:06.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:19:06.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26992/ceph-mon.a.asok config get mon_host 2024-09-18T11:19:06.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:388: TEST_profile_disallow_builtin_params_override: run_mgr td/mclock-config x 2024-09-18T11:19:06.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-18T11:19:06.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:19:06.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:19:06.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:19:06.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-18T11:19:06.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:19:07.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:19:07.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:19:07.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:19:07.393 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:19:07.394 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:07.394 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:07.394 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:19:07.394 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:19:07.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:19:07.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:390: TEST_profile_disallow_builtin_params_override: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-18T11:19:07.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-18T11:19:07.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:19:07.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:19:07.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:19:07.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-18T11:19:07.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-18T11:19:07.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:19:07.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:19:07.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:19:07.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-18T11:19:07.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-18T11:19:07.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:19:07.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:19:07.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-18T11:19:07.420 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:19:07.420 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:19:07.420 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:19:07.420 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:19:07.420 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:07.420 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:07.420 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:19:07.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:19:07.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:19:07.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:19:07.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:19:07.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-18T11:19:07.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-18T11:19:07.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:19:07.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:19:07.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:19:07.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:19:07.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:19:07.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-18T11:19:07.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-18T11:19:07.421 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:19:07.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1ed22c06-a0a7-461d-8f2c-372b5f72f4a7 2024-09-18T11:19:07.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1ed22c06-a0a7-461d-8f2c-372b5f72f4a7' 2024-09-18T11:19:07.422 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 1ed22c06-a0a7-461d-8f2c-372b5f72f4a7 2024-09-18T11:19:07.422 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:19:07.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQArt+pmHc62GRAAN/A+eILA2fPS187YCzh3Uw== 2024-09-18T11:19:07.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQArt+pmHc62GRAAN/A+eILA2fPS187YCzh3Uw=="}' 2024-09-18T11:19:07.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1ed22c06-a0a7-461d-8f2c-372b5f72f4a7 -i td/mclock-config/0/new.json 2024-09-18T11:19:07.704 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:19:07.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-18T11:19:07.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQArt+pmHc62GRAAN/A+eILA2fPS187YCzh3Uw== --osd-uuid 1ed22c06-a0a7-461d-8f2c-372b5f72f4a7 2024-09-18T11:19:07.738 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:07.736+0000 7f0410323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:07.749 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:07.747+0000 7f0410323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:07.750 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:07.749+0000 7f0410323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:07.751 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:07.749+0000 7f0410323440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:19:07.751 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:07.749+0000 7f0410323440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:19:07.751 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:07.749+0000 7f0410323440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:19:07.789 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:07.788+0000 7f0410323440 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-18T11:19:13.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-18T11:19:13.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:19:13.121 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:19:13.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:19:13.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:19:13.604 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:19:13.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:19:13.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-18T11:19:13.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:19:13.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:19:13.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:19:13.626 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:13.624+0000 7f11b2c0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:13.628 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:13.626+0000 7f11b2c0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:13.629 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:13.627+0000 7f11b2c0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:13.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:19:13.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:19:13.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:19:13.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:19:13.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:19:13.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:13.954 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:19:13.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:19:13.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:13.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:14.162 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:14.161+0000 7f11b2c0c440 -1 Falling back to public interface 2024-09-18T11:19:14.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:15.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:15.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:15.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:19:15.310 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:19:15.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:15.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:15.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:16.656 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:16.655+0000 7f11b2c0c440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:19:16.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:16.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:16.667 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:19:16.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:19:16.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:16.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:17.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:18.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:18.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:18.027 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:19:18.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:19:18.028 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:18.028 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:18.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:19.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:19.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:19.478 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:19:19.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:19:19.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:19.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:19.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:20.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:20.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:20.854 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:19:20.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:19:20.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:20.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:21.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:22.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:22.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:22.324 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:19:22.325 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:19:22.325 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:22.325 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:22.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:23.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:23.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:23.728 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:19:23.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:19:23.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:23.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:24.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:25.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:25.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:25.124 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:19:25.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:19:25.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:25.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:25.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:26.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:26.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:26.605 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:19:26.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:19:26.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:26.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:27.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:28.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:28.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:28.010 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:19:28.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:19:28.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:28.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:28.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:29.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:29.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:29.859 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:19:29.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:19:29.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:29.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:30.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:31.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:31.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:31.566 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:19:31.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:19:31.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:31.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:32.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:33.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:33.261 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:19:33.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:33.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:19:33.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:33.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:34.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:35.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:35.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:35.112 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:19:35.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:19:35.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:35.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:35.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:36.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:36.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:36.572 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:19:36.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:19:36.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:36.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:37.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:38.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:38.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:38.307 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:19:38.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:19:38.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:38.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:38.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:39.915 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:39.915 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:19:39.916 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:39.916 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:19:39.916 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:39.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:40.261 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/37754,v1:127.0.0.1:6803/37754] [v2:127.0.0.1:6804/37754,v1:127.0.0.1:6805/37754] exists,up 1ed22c06-a0a7-461d-8f2c-372b5f72f4a7 2024-09-18T11:19:40.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:19:40.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:19:40.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:19:40.262 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_profile 2024-09-18T11:19:40.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: local def_mclock_profile=balanced 2024-09-18T11:19:40.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:394: TEST_profile_disallow_builtin_params_override: test balanced = balanced 2024-09-18T11:19:40.614 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_profile 2024-09-18T11:19:40.614 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-18T11:19:40.614 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:19:40.614 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:19:40.614 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:19:40.614 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:40.614 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:40.615 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:19:40.615 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-18T11:19:40.615 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_profile 2024-09-18T11:19:40.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: local 'cur_mclock_profile="high_recovery_ops"' 2024-09-18T11:19:40.694 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: eval echo '"high_recovery_ops"' 2024-09-18T11:19:40.694 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: echo high_recovery_ops 2024-09-18T11:19:40.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: cur_mclock_profile=high_recovery_ops 2024-09-18T11:19:40.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:401: TEST_profile_disallow_builtin_params_override: test high_recovery_ops = high_recovery_ops 2024-09-18T11:19:40.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-09-18T11:19:40.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: declare -a options 2024-09-18T11:19:40.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:406: TEST_profile_disallow_builtin_params_override: local retries=10 2024-09-18T11:19:40.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:407: TEST_profile_disallow_builtin_params_override: local errors=0 2024-09-18T11:19:40.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-09-18T11:19:40.696 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-09-18T11:19:40.696 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-09-18T11:19:40.696 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-18T11:19:40.696 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:19:40.696 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:19:40.696 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:19:40.696 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:40.697 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:40.697 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:19:40.697 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-18T11:19:40.697 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-18T11:19:40.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.700000 2024-09-18T11:19:40.774 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.700000 + 0.1' 2024-09-18T11:19:40.774 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-18T11:19:40.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.800000 2024-09-18T11:19:40.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_background_recovery_res .800000 2024-09-18T11:19:40.873 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:19:40.873 INFO:tasks.workunit.client.0.smithi157.stdout: "success": "osd_mclock_scheduler_background_recovery_res = '' " 2024-09-18T11:19:40.873 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:19:40.882 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-09-18T11:19:40.883 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-09-18T11:19:40.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-09-18T11:19:40.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-09-18T11:19:40.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-09-18T11:19:42.885 INFO:tasks.workunit.client.0.smithi157.stdout:Check configuration values - Attempt#: 0 2024-09-18T11:19:42.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-09-18T11:19:42.886 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-09-18T11:19:43.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-09-18T11:19:43.273 INFO:tasks.workunit.client.0.smithi157.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-09-18T11:19:43.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-09-18T11:19:43.274 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .800000' 2024-09-18T11:19:43.274 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-18T11:19:43.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-09-18T11:19:43.275 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-09-18T11:19:43.275 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_background_recovery_res 2024-09-18T11:19:43.275 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-09-18T11:19:43.275 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-09-18T11:19:43.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.700000 2024-09-18T11:19:43.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-09-18T11:19:43.622 INFO:tasks.workunit.client.0.smithi157.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-09-18T11:19:43.623 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 == .800000' 2024-09-18T11:19:43.623 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-18T11:19:43.623 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 != 0.700000' 2024-09-18T11:19:43.623 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-18T11:19:43.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-18T11:19:43.626 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-09-18T11:19:43.626 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-09-18T11:19:43.626 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-18T11:19:43.626 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:19:43.626 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:19:43.626 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:19:43.627 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:43.627 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:43.627 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:19:43.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-18T11:19:43.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-18T11:19:43.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.700000 2024-09-18T11:19:43.705 INFO:tasks.workunit.client.0.smithi157.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-09-18T11:19:43.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-09-18T11:19:43.705 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 == .800000' 2024-09-18T11:19:43.705 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-18T11:19:43.706 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 != 0.700000' 2024-09-18T11:19:43.706 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-18T11:19:43.707 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-18T11:19:43.707 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-09-18T11:19:43.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-09-18T11:19:43.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-09-18T11:19:43.709 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-09-18T11:19:43.709 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-09-18T11:19:43.709 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-18T11:19:43.709 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:19:43.709 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:19:43.709 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:19:43.710 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:43.710 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:43.710 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:19:43.710 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-18T11:19:43.710 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-18T11:19:43.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.300000 2024-09-18T11:19:43.788 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.300000 + 0.1' 2024-09-18T11:19:43.789 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-18T11:19:43.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.400000 2024-09-18T11:19:43.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_client_res .400000 2024-09-18T11:19:43.887 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:19:43.887 INFO:tasks.workunit.client.0.smithi157.stdout: "success": "osd_mclock_scheduler_client_res = '' " 2024-09-18T11:19:43.887 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:19:43.896 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-09-18T11:19:43.897 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-09-18T11:19:43.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-09-18T11:19:43.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-09-18T11:19:43.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-09-18T11:19:45.899 INFO:tasks.workunit.client.0.smithi157.stdout:Check configuration values - Attempt#: 0 2024-09-18T11:19:45.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-09-18T11:19:45.900 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-18T11:19:46.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-09-18T11:19:46.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-09-18T11:19:46.262 INFO:tasks.workunit.client.0.smithi157.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-09-18T11:19:46.262 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .400000' 2024-09-18T11:19:46.263 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-18T11:19:46.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-09-18T11:19:46.264 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-09-18T11:19:46.264 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_client_res 2024-09-18T11:19:46.264 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-09-18T11:19:46.264 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-09-18T11:19:46.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.300000 2024-09-18T11:19:46.650 INFO:tasks.workunit.client.0.smithi157.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-09-18T11:19:46.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-09-18T11:19:46.651 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 == .400000' 2024-09-18T11:19:46.651 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-18T11:19:46.653 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 != 0.300000' 2024-09-18T11:19:46.653 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-18T11:19:46.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-18T11:19:46.655 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-09-18T11:19:46.655 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-09-18T11:19:46.655 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-18T11:19:46.655 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:19:46.655 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:19:46.655 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:19:46.655 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:46.655 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:46.655 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:19:46.656 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-18T11:19:46.656 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-18T11:19:46.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.300000 2024-09-18T11:19:46.734 INFO:tasks.workunit.client.0.smithi157.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-09-18T11:19:46.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-09-18T11:19:46.734 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 == .400000' 2024-09-18T11:19:46.735 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-18T11:19:46.736 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 != 0.300000' 2024-09-18T11:19:46.736 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-18T11:19:46.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-18T11:19:46.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-09-18T11:19:46.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-09-18T11:19:46.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:459: TEST_profile_disallow_builtin_params_override: teardown td/mclock-config 2024-09-18T11:19:46.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:19:46.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:19:46.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:19:46.738 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:19:46.738 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:19:46.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:19:46.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:19:46.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:19:46.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:19:46.843 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:19:46.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:19:46.844 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:19:46.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:19:46.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:19:46.846 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:19:46.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:19:46.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:19:46.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:19:46.847 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:19:46.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:19:46.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:19:46.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:19:46.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:19:46.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:46.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:46.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:19:46.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:19:46.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:19:46.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-18T11:19:46.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:19:46.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:19:46.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:19:46.856 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:19:46.856 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:19:46.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:19:46.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:19:46.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:19:46.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:19:46.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:19:46.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:19:46.858 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:19:46.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:19:46.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:19:46.860 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:19:46.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:19:46.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:19:46.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:19:46.861 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:19:46.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:19:46.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:19:46.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:19:46.864 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:19:46.864 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:46.864 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:46.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:19:46.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:19:46.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:19:46.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-18T11:19:46.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-18T11:19:46.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-18T11:19:46.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-18T11:19:46.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:19:46.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:19:46.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:19:46.866 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:19:46.866 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:19:46.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:19:46.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:19:46.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:19:46.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:19:46.867 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:19:46.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:19:46.868 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:19:46.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:19:46.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:19:46.870 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:19:46.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:19:46.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:19:46.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:19:46.871 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:19:46.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:19:46.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:19:46.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:19:46.874 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:19:46.874 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:46.874 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:46.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:19:46.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:19:46.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:19:46.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-18T11:19:46.877 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:19:46.877 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:46.877 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:46.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26992 2024-09-18T11:19:46.879 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:19:46.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:19:46.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:19:46.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-18T11:19:46.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2024-09-18T11:19:46.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2024-09-18T11:19:46.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2024-09-18T11:19:46.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-18T11:19:46.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-18T11:19:46.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:19:46.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:19:46.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:19:46.880 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:19:46.880 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:19:46.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:19:46.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:19:46.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:19:46.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:19:46.880 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:19:46.881 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:19:46.881 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:19:46.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:19:46.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:19:46.882 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:19:46.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:19:46.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:19:46.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:19:46.884 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:19:46.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:19:46.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:19:46.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:19:46.886 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:19:46.886 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:46.886 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:46.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:19:46.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:19:46.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:19:46.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-18T11:19:46.888 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:19:46.889 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:46.889 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:46.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26992 2024-09-18T11:19:46.890 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:19:46.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:19:46.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:19:46.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-18T11:19:46.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2024-09-18T11:19:46.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-18T11:19:46.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:19:46.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:19:46.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:19:46.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-18T11:19:46.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-18T11:19:47.244 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:19:47.244 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:19:47.244 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:19:47.245 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:19:47.245 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:47.245 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:47.245 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:19:47.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:19:47.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:19:47.490 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:19:47.490 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:19:47.490 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:19:47.490 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:19:47.490 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:19:47.490 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:19:47.490 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:19:47.490 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:19:47.490 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:19:47.490 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:47.490 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:47.490 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-mon.a.asok 2024-09-18T11:19:47.490 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:19:47.491 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26992/ceph-mon.a.asok config get fsid 2024-09-18T11:19:47.564 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:19:47.564 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:19:47.564 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:19:47.564 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:19:47.564 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:19:47.564 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:19:47.564 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:19:47.564 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:19:47.564 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:19:47.565 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:47.565 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:47.565 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-mon.a.asok 2024-09-18T11:19:47.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:19:47.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26992/ceph-mon.a.asok config get mon_host 2024-09-18T11:19:47.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-09-18T11:19:47.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-18T11:19:47.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:19:47.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:19:47.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:19:47.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-18T11:19:47.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:19:48.475 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:19:48.475 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:19:48.475 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:19:48.475 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:19:48.475 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:48.476 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:48.476 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:19:48.476 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:19:48.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:19:48.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-18T11:19:48.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-18T11:19:48.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:19:48.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:19:48.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:19:48.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-18T11:19:48.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-18T11:19:48.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:19:48.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:19:48.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:19:48.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-18T11:19:48.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-18T11:19:48.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:19:48.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:19:48.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-18T11:19:48.502 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:19:48.503 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:19:48.503 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:19:48.503 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:19:48.503 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:48.503 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:19:48.503 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:19:48.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' 2024-09-18T11:19:48.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:19:48.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:19:48.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:19:48.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-18T11:19:48.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-18T11:19:48.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:19:48.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:19:48.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:19:48.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:19:48.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:19:48.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-18T11:19:48.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-18T11:19:48.504 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:19:48.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b60084e7-566e-4201-a8f2-6b148dcf9a83 2024-09-18T11:19:48.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b60084e7-566e-4201-a8f2-6b148dcf9a83' 2024-09-18T11:19:48.505 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 b60084e7-566e-4201-a8f2-6b148dcf9a83 2024-09-18T11:19:48.505 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:19:48.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBUt+pmyLOiHhAApqzuYWJct4fBuyCKbzQxew== 2024-09-18T11:19:48.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBUt+pmyLOiHhAApqzuYWJct4fBuyCKbzQxew=="}' 2024-09-18T11:19:48.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b60084e7-566e-4201-a8f2-6b148dcf9a83 -i td/mclock-config/0/new.json 2024-09-18T11:19:48.835 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:19:48.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-18T11:19:48.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBUt+pmyLOiHhAApqzuYWJct4fBuyCKbzQxew== --osd-uuid b60084e7-566e-4201-a8f2-6b148dcf9a83 2024-09-18T11:19:48.868 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:48.867+0000 7f8362008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:48.871 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:48.870+0000 7f8362008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:48.872 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:48.871+0000 7f8362008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:48.873 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:48.871+0000 7f8362008440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:19:48.873 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:48.871+0000 7f8362008440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:19:48.873 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:48.871+0000 7f8362008440 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-18T11:19:48.912 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:48.911+0000 7f8362008440 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-18T11:19:54.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-18T11:19:54.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:19:54.233 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:19:54.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:19:54.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:19:54.718 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:19:54.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:19:54.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8ed51677-9cce-44c2-9bf2-cf2d1531f877 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26992/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-18T11:19:54.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:19:54.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:19:54.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:19:54.740 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:54.739+0000 7fc9a9608440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:54.768 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:54.766+0000 7fc9a9608440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:54.768 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:54.767+0000 7fc9a9608440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:55.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:19:55.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:19:55.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:19:55.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:19:55.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:19:55.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:55.117 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:19:55.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:19:55.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:55.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:55.297 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:55.296+0000 7fc9a9608440 -1 Falling back to public interface 2024-09-18T11:19:55.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:56.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:56.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:56.474 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:19:56.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:19:56.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:56.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:56.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:57.779 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:19:57.778+0000 7fc9a9608440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:19:57.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:57.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:57.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:19:57.825 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:19:57.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:57.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:58.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:59.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:59.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:59.203 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:19:59.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:19:59.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:59.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:19:59.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:00.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:00.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:00.634 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:20:00.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:20:00.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:00.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:01.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:02.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:02.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:02.023 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:20:02.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:20:02.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:02.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:02.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:03.463 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:03.463 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:03.463 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:20:03.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:20:03.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:03.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:03.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:04.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:04.825 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:20:04.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:04.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:20:04.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:04.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:05.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:06.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:06.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:06.198 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:20:06.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:20:06.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:06.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:06.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:07.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:07.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:07.637 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:20:07.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:20:07.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:07.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:08.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:09.034 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:09.034 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:09.034 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:20:09.034 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:20:09.035 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:09.035 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:09.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:10.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:10.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:10.435 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:20:10.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:20:10.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:10.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:10.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:11.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:11.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:11.937 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:20:11.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:20:11.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:11.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:12.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:13.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:13.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:13.665 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:20:13.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:20:13.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:13.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:14.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:15.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:15.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:15.206 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:20:15.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:20:15.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:15.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:15.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:16.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:16.970 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:20:16.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:16.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:20:16.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:16.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:17.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:18.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:18.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:18.651 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:20:18.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:20:18.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:18.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:19.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:20.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:20.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:20.436 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:20:20.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:20:20.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:20.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:20.793 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/40047,v1:127.0.0.1:6803/40047] [v2:127.0.0.1:6804/40047,v1:127.0.0.1:6805/40047] exists,up b60084e7-566e-4201-a8f2-6b148dcf9a83 2024-09-18T11:20:20.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:20:20.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:20:20.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:20:20.795 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-18T11:20:20.795 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:20:20.795 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:20:20.795 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:20:20.795 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:20:20.795 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:20:20.795 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:20:20.796 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-18T11:20:20.796 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-18T11:20:20.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2024-09-18T11:20:20.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2024-09-18T11:20:20.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2024-09-18T11:20:20.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-09-18T11:20:21.458 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-09-18T11:20:23.461 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-18T11:20:23.461 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:20:23.461 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:20:23.461 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:20:23.461 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:20:23.461 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:20:23.461 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:20:23.461 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-18T11:20:23.461 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-18T11:20:23.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2024-09-18T11:20:23.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2024-09-18T11:20:23.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-09-18T11:20:23.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-09-18T11:20:24.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-09-18T11:20:26.409 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-18T11:20:26.410 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:20:26.410 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:20:26.410 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:20:26.410 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:20:26.410 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:20:26.410 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26992/ceph-osd.0.asok 2024-09-18T11:20:26.410 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-18T11:20:26.411 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26992/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-18T11:20:26.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2024-09-18T11:20:26.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2024-09-18T11:20:26.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2024-09-18T11:20:26.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:20:26.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:20:26.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:20:26.501 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:20:26.501 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:20:26.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:20:26.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:20:26.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:20:26.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:20:26.607 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:20:26.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:20:26.608 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:20:26.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:20:26.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:20:26.610 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:20:26.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:20:26.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:20:26.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:20:26.611 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:20:26.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:20:26.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:20:26.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:20:26.622 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:20:26.622 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:20:26.622 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:20:26.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:20:26.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:20:26.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:20:26.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-18T11:20:26.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:20:26.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:20:26.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:20:26.623 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:20:26.623 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:20:26.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:20:26.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:20:26.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:20:26.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:20:26.624 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:20:26.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:20:26.624 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:20:26.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:20:26.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:20:26.625 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:20:26.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:20:26.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:20:26.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:20:26.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:20:26.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:20:26.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:20:26.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:20:26.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:20:26.630 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:20:26.630 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:20:26.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:20:26.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:20:26.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:20:26.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T11:20:26.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/mclock-config 0 2024-09-18T11:20:26.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-18T11:20:26.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T11:20:26.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-18T11:20:26.632 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:20:26.632 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:20:26.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:20:26.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:20:26.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:20:26.633 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:20:26.633 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:20:26.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:20:26.634 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:20:26.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:20:26.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:20:26.636 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:20:26.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:20:26.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:20:26.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:20:26.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:20:26.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:20:26.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T11:20:26.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-18T11:20:26.640 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:20:26.640 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:20:26.640 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26992 2024-09-18T11:20:26.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26992 2024-09-18T11:20:26.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:20:26.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:20:26.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T11:20:26.642 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T11:20:26.642 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T11:20:26.708 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2024-09-18T11:20:26.709 DEBUG:teuthology.orchestra.run.smithi157:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2024-09-18T11:20:26.769 INFO:tasks.workunit.client.0.smithi157.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T11:20:26.775 INFO:tasks.workunit.client.0.smithi157.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T11:20:26.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T11:20:26.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T11:20:26.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T11:20:26.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T11:20:26.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T11:20:26.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T11:20:26.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T11:20:26.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T11:20:26.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/network-ping 2024-09-18T11:20:26.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2024-09-18T11:20:26.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2024-09-18T11:20:26.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2024-09-18T11:20:26.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2024-09-18T11:20:26.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2024-09-18T11:20:26.777 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2024-09-18T11:20:26.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none ' 2024-09-18T11:20:26.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-18T11:20:26.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2024-09-18T11:20:26.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2024-09-18T11:20:26.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2024-09-18T11:20:26.777 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2024-09-18T11:20:26.777 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T11:20:26.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2024-09-18T11:20:26.779 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_network_ping_test2' 2024-09-18T11:20:26.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-09-18T11:20:26.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-09-18T11:20:26.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-09-18T11:20:26.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-09-18T11:20:26.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-18T11:20:26.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:20:26.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-18T11:20:26.779 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:20:26.779 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:20:26.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:20:26.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:20:26.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:20:26.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:20:26.780 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:20:26.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:20:26.780 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:20:26.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:20:26.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:20:26.781 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:20:26.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:20:26.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:20:26.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:20:26.782 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:20:26.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:20:26.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:20:26.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-18T11:20:26.785 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:20:26.785 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:20:26.785 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:20:26.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.41135 2024-09-18T11:20:26.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:20:26.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:20:26.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-09-18T11:20:26.787 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:20:26.787 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:20:26.787 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:20:26.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.41135 2024-09-18T11:20:26.789 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:20:26.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T11:20:26.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T11:20:26.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:20:26.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-09-18T11:20:26.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2024-09-18T11:20:26.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2024-09-18T11:20:26.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2024-09-18T11:20:26.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-09-18T11:20:26.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:20:26.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:20:26.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:20:26.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-09-18T11:20:26.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-09-18T11:20:27.157 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:20:27.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:20:27.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:20:27.158 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:20:27.158 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:20:27.158 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:20:27.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41135/$cluster-$name.asok' 2024-09-18T11:20:27.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:20:27.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:20:27.411 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:20:27.411 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:20:27.411 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:20:27.411 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:20:27.411 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:20:27.411 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:20:27.411 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:20:27.411 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:20:27.411 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:20:27.411 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:20:27.411 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:20:27.411 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-mon.a.asok 2024-09-18T11:20:27.412 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:20:27.412 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.41135/ceph-mon.a.asok config get fsid 2024-09-18T11:20:27.487 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:20:27.487 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:20:27.488 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:20:27.488 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:20:27.488 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:20:27.488 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:20:27.488 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:20:27.488 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:20:27.488 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:20:27.488 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:20:27.488 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:20:27.488 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-mon.a.asok 2024-09-18T11:20:27.488 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:20:27.488 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.41135/ceph-mon.a.asok config get mon_host 2024-09-18T11:20:27.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2024-09-18T11:20:27.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-09-18T11:20:27.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:20:27.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:20:27.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:20:27.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-09-18T11:20:27.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:20:28.403 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:20:28.403 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:20:28.403 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:20:28.404 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:20:28.404 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:20:28.404 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:20:28.404 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41135/$cluster-$name.asok' 2024-09-18T11:20:28.404 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:20:28.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:20:28.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2024-09-18T11:20:28.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-18T11:20:28.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:20:28.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:20:28.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:20:28.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-09-18T11:20:28.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-18T11:20:28.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:20:28.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:20:28.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:20:28.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-09-18T11:20:28.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-09-18T11:20:28.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:20:28.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:20:28.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-18T11:20:28.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:20:28.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:20:28.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:20:28.432 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:20:28.432 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:20:28.432 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:20:28.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41135/$cluster-$name.asok' 2024-09-18T11:20:28.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' 2024-09-18T11:20:28.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:20:28.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:20:28.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:20:28.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-18T11:20:28.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-18T11:20:28.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:20:28.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:20:28.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:20:28.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:20:28.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:20:28.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:20:28.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-09-18T11:20:28.433 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:20:28.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fc4e6ed1-ef30-494e-8440-b91377879142 2024-09-18T11:20:28.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fc4e6ed1-ef30-494e-8440-b91377879142' 2024-09-18T11:20:28.434 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 fc4e6ed1-ef30-494e-8440-b91377879142 2024-09-18T11:20:28.434 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:20:28.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB8t+pmz6eCGhAA8XZBnr5LpH1+8fC+7EavVA== 2024-09-18T11:20:28.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB8t+pmz6eCGhAA8XZBnr5LpH1+8fC+7EavVA=="}' 2024-09-18T11:20:28.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fc4e6ed1-ef30-494e-8440-b91377879142 -i td/network-ping/0/new.json 2024-09-18T11:20:28.765 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:20:28.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-09-18T11:20:28.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8t+pmz6eCGhAA8XZBnr5LpH1+8fC+7EavVA== --osd-uuid fc4e6ed1-ef30-494e-8440-b91377879142 2024-09-18T11:20:28.805 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:20:28.803+0000 7f889ce51440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:20:28.810 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:20:28.809+0000 7f889ce51440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:20:28.811 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:20:28.810+0000 7f889ce51440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:20:28.812 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:20:28.810+0000 7f889ce51440 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-18T11:20:28.812 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:20:28.810+0000 7f889ce51440 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-18T11:20:28.812 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:20:28.810+0000 7f889ce51440 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-18T11:20:28.851 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:20:28.849+0000 7f889ce51440 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-09-18T11:20:34.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-09-18T11:20:34.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:20:34.163 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:20:34.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:20:34.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:20:34.659 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:20:34.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:20:34.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:20:34.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:20:34.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:20:34.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:20:34.683 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:20:34.682+0000 7f14a4a16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:20:34.685 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:20:34.684+0000 7f14a4a16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:20:34.686 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:20:34.685+0000 7f14a4a16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:20:35.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:20:35.019 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:20:35.019 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:20:35.019 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:20:35.019 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:20:35.019 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:35.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:20:35.020 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:20:35.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:35.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:35.208 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:20:35.207+0000 7f14a4a16440 -1 Falling back to public interface 2024-09-18T11:20:35.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:36.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:36.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:36.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:20:36.374 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:20:36.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:36.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:36.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:37.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:37.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:37.730 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:20:37.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:20:37.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:37.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:37.794 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:20:37.792+0000 7f14a4a16440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:20:38.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:39.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:39.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:39.094 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:20:39.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:20:39.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:39.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:39.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:40.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:40.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:40.545 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:20:40.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:20:40.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:40.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:40.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:41.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:41.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:41.933 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:20:41.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:20:41.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:41.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:42.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:43.335 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:20:43.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:43.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:43.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:20:43.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:43.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:43.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:44.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:44.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:44.768 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:20:44.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:20:44.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:44.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:45.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:46.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:46.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:46.172 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:20:46.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:20:46.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:46.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:46.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:47.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:47.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:47.692 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:20:47.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:20:47.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:47.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:48.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:49.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:49.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:49.057 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:20:49.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:20:49.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:49.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:49.531 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:50.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:50.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:50.532 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:20:50.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:20:50.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:50.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:51.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:52.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:52.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:52.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:20:52.217 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:20:52.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:52.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:53.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:54.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:54.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:54.058 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:20:54.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:20:54.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:54.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:54.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:55.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:55.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:55.504 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:20:55.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:20:55.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:55.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:56.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:57.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:57.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:57.263 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:20:57.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:20:57.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:57.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:57.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:58.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:58.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:58.960 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:20:58.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:20:58.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:58.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:20:59.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:00.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:00.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:00.753 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:21:00.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:21:00.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:00.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:21:01.110 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/42237,v1:127.0.0.1:6803/42237] [v2:127.0.0.1:6804/42237,v1:127.0.0.1:6805/42237] exists,up fc4e6ed1-ef30-494e-8440-b91377879142 2024-09-18T11:21:01.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:21:01.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:21:01.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:21:01.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2024-09-18T11:21:01.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-18T11:21:01.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:21:01.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:21:01.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:21:01.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-09-18T11:21:01.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-18T11:21:01.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:21:01.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:21:01.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:21:01.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-09-18T11:21:01.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-09-18T11:21:01.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:21:01.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:21:01.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-18T11:21:01.116 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:21:01.116 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:21:01.116 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:21:01.116 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:21:01.116 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:21:01.117 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:21:01.117 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41135/$cluster-$name.asok' 2024-09-18T11:21:01.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' 2024-09-18T11:21:01.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:21:01.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:21:01.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:21:01.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-18T11:21:01.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-18T11:21:01.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:21:01.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:21:01.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:21:01.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:21:01.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:21:01.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:21:01.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-09-18T11:21:01.118 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:21:01.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8c295c1c-1352-4588-b426-58747522eac4 2024-09-18T11:21:01.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8c295c1c-1352-4588-b426-58747522eac4' 2024-09-18T11:21:01.118 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 8c295c1c-1352-4588-b426-58747522eac4 2024-09-18T11:21:01.118 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:21:01.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCdt+pmrFGYBxAAk9TUJ+k58zMB5XGZtyQLhw== 2024-09-18T11:21:01.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCdt+pmrFGYBxAAk9TUJ+k58zMB5XGZtyQLhw=="}' 2024-09-18T11:21:01.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8c295c1c-1352-4588-b426-58747522eac4 -i td/network-ping/1/new.json 2024-09-18T11:21:01.598 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:21:01.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-09-18T11:21:01.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdt+pmrFGYBxAAk9TUJ+k58zMB5XGZtyQLhw== --osd-uuid 8c295c1c-1352-4588-b426-58747522eac4 2024-09-18T11:21:01.635 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:01.633+0000 7f9ffef1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:01.636 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:01.635+0000 7f9ffef1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:01.638 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:01.636+0000 7f9ffef1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:01.638 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:01.636+0000 7f9ffef1f440 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-18T11:21:01.638 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:01.636+0000 7f9ffef1f440 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-18T11:21:01.638 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:01.636+0000 7f9ffef1f440 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-18T11:21:01.719 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:01.717+0000 7f9ffef1f440 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-09-18T11:21:07.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-09-18T11:21:07.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:21:07.876 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T11:21:07.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:21:07.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:21:08.375 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T11:21:08.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:21:08.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:21:08.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:21:08.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:21:08.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:21:08.398 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:08.397+0000 7f257d12d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:08.400 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:08.399+0000 7f257d12d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:08.401 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:08.400+0000 7f257d12d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:08.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:21:08.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:21:08.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:21:08.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:21:08.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:21:08.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:08.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:21:08.766 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:21:08.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:08.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:08.918 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:08.917+0000 7f257d12d440 -1 Falling back to public interface 2024-09-18T11:21:09.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:10.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:10.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:10.153 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:21:10.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:21:10.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:10.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:10.553 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:11.400 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:11.398+0000 7f257d12d440 -1 osd.1 0 log_to_monitors true 2024-09-18T11:21:11.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:11.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:11.555 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:21:11.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:21:11.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:11.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:12.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:13.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:13.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:13.178 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:21:13.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:21:13.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:13.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:13.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:14.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:14.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:14.658 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:21:14.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:21:14.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:14.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:15.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:16.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:16.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:16.177 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:21:16.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:21:16.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:16.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:16.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:17.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:17.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:17.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:21:17.543 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:21:17.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:17.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:17.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:18.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:18.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:18.950 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:21:18.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:21:18.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:18.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:19.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:20.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:20.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:20.315 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:21:20.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:21:20.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:20.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:20.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:21.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:21.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:21.778 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:21:21.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:21:21.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:21.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:22.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:23.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:23.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:23.244 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:21:23.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:21:23.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:23.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:23.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:24.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:24.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:24.609 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:21:24.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:21:24.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:24.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:25.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:26.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:26.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:26.506 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:21:26.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:21:26.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:26.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:26.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:27.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:27.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:27.905 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:21:27.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:21:27.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:27.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:28.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:29.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:29.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:29.421 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:21:29.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:21:29.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:29.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:30.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:31.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:31.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:31.136 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:21:31.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:21:31.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:31.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:31.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:32.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:32.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:32.854 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:21:32.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:21:32.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:32.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:33.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:34.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:34.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:34.665 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:21:34.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:21:34.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:34.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:21:35.124 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/43992,v1:127.0.0.1:6811/43992] [v2:127.0.0.1:6812/43992,v1:127.0.0.1:6813/43992] exists,up 8c295c1c-1352-4588-b426-58747522eac4 2024-09-18T11:21:35.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:21:35.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:21:35.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:21:35.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2024-09-18T11:21:35.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-18T11:21:35.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:21:35.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:21:35.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:21:35.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-09-18T11:21:35.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-18T11:21:35.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:21:35.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:21:35.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:21:35.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-09-18T11:21:35.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-09-18T11:21:35.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:21:35.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:21:35.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-18T11:21:35.129 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:21:35.129 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:21:35.129 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:21:35.129 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:21:35.129 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:21:35.129 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:21:35.129 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41135/$cluster-$name.asok' 2024-09-18T11:21:35.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' 2024-09-18T11:21:35.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:21:35.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:21:35.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:21:35.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-18T11:21:35.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-18T11:21:35.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:21:35.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:21:35.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:21:35.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:21:35.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:21:35.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:21:35.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-09-18T11:21:35.131 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:21:35.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=77c1d60c-6a86-4849-bcf5-ad1bc8fb7fe9 2024-09-18T11:21:35.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 77c1d60c-6a86-4849-bcf5-ad1bc8fb7fe9' 2024-09-18T11:21:35.131 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 77c1d60c-6a86-4849-bcf5-ad1bc8fb7fe9 2024-09-18T11:21:35.132 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:21:35.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/t+pmktV3CBAAPxuqoDwVMLMq2jzzH/3XHw== 2024-09-18T11:21:35.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/t+pmktV3CBAAPxuqoDwVMLMq2jzzH/3XHw=="}' 2024-09-18T11:21:35.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 77c1d60c-6a86-4849-bcf5-ad1bc8fb7fe9 -i td/network-ping/2/new.json 2024-09-18T11:21:35.612 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:21:35.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-09-18T11:21:35.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/t+pmktV3CBAAPxuqoDwVMLMq2jzzH/3XHw== --osd-uuid 77c1d60c-6a86-4849-bcf5-ad1bc8fb7fe9 2024-09-18T11:21:35.648 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:35.647+0000 7f365d234440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:35.650 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:35.648+0000 7f365d234440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:35.651 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:35.649+0000 7f365d234440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:35.651 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:35.650+0000 7f365d234440 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-18T11:21:35.651 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:35.650+0000 7f365d234440 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-18T11:21:35.651 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:35.650+0000 7f365d234440 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-18T11:21:35.727 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:35.726+0000 7f365d234440 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-09-18T11:21:41.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-09-18T11:21:41.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:21:41.610 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-18T11:21:41.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:21:41.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:21:42.113 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-18T11:21:42.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:21:42.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:21:42.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:21:42.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:21:42.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:21:42.136 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:42.134+0000 7f8bb771c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:42.138 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:42.137+0000 7f8bb771c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:42.139 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:42.138+0000 7f8bb771c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:42.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:21:42.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:21:42.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:21:42.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:21:42.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:21:42.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:42.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:21:42.507 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:21:42.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:42.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:21:42.656 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:42.654+0000 7f8bb771c440 -1 Falling back to public interface 2024-09-18T11:21:42.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:43.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:43.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:43.862 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:21:43.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:21:43.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:43.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:21:44.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:45.224 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:21:45.221+0000 7f8bb771c440 -1 osd.2 0 log_to_monitors true 2024-09-18T11:21:45.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:45.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:45.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:21:45.225 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:21:45.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:45.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:21:45.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:46.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:46.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:46.675 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:21:46.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:21:46.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:46.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:21:47.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:48.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:48.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:48.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:21:48.082 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:21:48.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:48.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:21:48.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:49.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:49.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:49.452 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:21:49.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:21:49.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:49.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:21:49.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:50.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:50.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:50.813 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:21:50.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:21:50.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:50.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:21:51.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:52.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:52.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:52.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:21:52.224 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:21:52.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:52.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:21:52.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:53.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:53.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:53.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:21:53.589 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:21:53.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:53.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:21:54.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:55.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:55.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:55.013 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:21:55.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:21:55.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:55.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:21:55.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:56.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:56.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:56.377 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:21:56.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:21:56.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:56.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:21:56.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:57.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:57.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:57.808 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:21:57.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:21:57.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:57.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:21:58.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:59.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:59.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:59.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:21:59.645 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:21:59.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:59.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:22:00.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:01.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:01.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:01.355 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:22:01.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:22:01.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:01.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:22:02.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:03.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:03.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:03.126 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:22:03.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:22:03.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:03.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:22:03.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:04.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:04.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:04.945 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:22:04.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:22:04.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:04.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:22:05.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:06.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:06.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:06.786 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:22:06.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:22:06.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:06.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:22:07.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:08.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:08.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:08.660 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:22:08.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:22:08.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:08.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:22:09.054 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/45747,v1:127.0.0.1:6819/45747] [v2:127.0.0.1:6820/45747,v1:127.0.0.1:6821/45747] exists,up 77c1d60c-6a86-4849-bcf5-ad1bc8fb7fe9 2024-09-18T11:22:09.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:22:09.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:22:09.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:22:09.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2024-09-18T11:22:14.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2024-09-18T11:22:14.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-18T11:22:14.546 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'foo' already exists 2024-09-18T11:22:14.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:22:15.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-09-18T11:22:15.643 INFO:tasks.workunit.client.0.smithi157.stdout:hints = 1 2024-09-18T11:22:15.643 INFO:tasks.workunit.client.0.smithi157.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-09-18T11:22:15.643 INFO:tasks.workunit.client.0.smithi157.stdout:Object prefix: benchmark_data_smithi157_46687 2024-09-18T11:22:15.643 INFO:tasks.workunit.client.0.smithi157.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:22:15.643 INFO:tasks.workunit.client.0.smithi157.stdout: 0 0 0 0 0 0 - 0 2024-09-18T11:22:16.643 INFO:tasks.workunit.client.0.smithi157.stdout: 1 16 63 47 0.183573 0.183594 0.259953 0.304803 2024-09-18T11:22:17.643 INFO:tasks.workunit.client.0.smithi157.stdout: 2 16 121 105 0.20506 0.226562 0.239393 0.279765 2024-09-18T11:22:18.643 INFO:tasks.workunit.client.0.smithi157.stdout: 3 16 185 169 0.220036 0.25 0.293632 0.265143 2024-09-18T11:22:19.643 INFO:tasks.workunit.client.0.smithi157.stdout: 4 16 249 233 0.227518 0.25 0.270802 0.263964 2024-09-18T11:22:20.643 INFO:tasks.workunit.client.0.smithi157.stdout: 5 16 313 297 0.232008 0.25 0.258219 0.259031 2024-09-18T11:22:21.644 INFO:tasks.workunit.client.0.smithi157.stdout: 6 16 384 368 0.239558 0.277344 0.250088 0.256406 2024-09-18T11:22:22.644 INFO:tasks.workunit.client.0.smithi157.stdout: 7 16 448 432 0.241046 0.25 0.264501 0.254392 2024-09-18T11:22:23.644 INFO:tasks.workunit.client.0.smithi157.stdout: 8 16 512 496 0.242161 0.25 0.25143 0.253691 2024-09-18T11:22:24.644 INFO:tasks.workunit.client.0.smithi157.stdout: 9 16 576 560 0.243028 0.25 0.266949 0.252133 2024-09-18T11:22:25.644 INFO:tasks.workunit.client.0.smithi157.stdout: 10 16 648 632 0.246846 0.28125 0.242917 0.251415 2024-09-18T11:22:26.644 INFO:tasks.workunit.client.0.smithi157.stdout:Total time run: 10.2386 2024-09-18T11:22:26.645 INFO:tasks.workunit.client.0.smithi157.stdout:Total writes made: 648 2024-09-18T11:22:26.645 INFO:tasks.workunit.client.0.smithi157.stdout:Write size: 4096 2024-09-18T11:22:26.645 INFO:tasks.workunit.client.0.smithi157.stdout:Object size: 4096 2024-09-18T11:22:26.645 INFO:tasks.workunit.client.0.smithi157.stdout:Bandwidth (MB/sec): 0.247225 2024-09-18T11:22:26.645 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev Bandwidth: 0.0270508 2024-09-18T11:22:26.645 INFO:tasks.workunit.client.0.smithi157.stdout:Max bandwidth (MB/sec): 0.28125 2024-09-18T11:22:26.645 INFO:tasks.workunit.client.0.smithi157.stdout:Min bandwidth (MB/sec): 0.183594 2024-09-18T11:22:26.645 INFO:tasks.workunit.client.0.smithi157.stdout:Average IOPS: 63 2024-09-18T11:22:26.645 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev IOPS: 6.92499 2024-09-18T11:22:26.645 INFO:tasks.workunit.client.0.smithi157.stdout:Max IOPS: 72 2024-09-18T11:22:26.645 INFO:tasks.workunit.client.0.smithi157.stdout:Min IOPS: 47 2024-09-18T11:22:26.645 INFO:tasks.workunit.client.0.smithi157.stdout:Average Latency(s): 0.251121 2024-09-18T11:22:26.646 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev Latency(s): 0.0311977 2024-09-18T11:22:26.646 INFO:tasks.workunit.client.0.smithi157.stdout:Max latency(s): 0.477791 2024-09-18T11:22:26.646 INFO:tasks.workunit.client.0.smithi157.stdout:Min latency(s): 0.207297 2024-09-18T11:22:26.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2024-09-18T11:22:36.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2024-09-18T11:22:36.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:22:36.762 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:22:37.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:22:37.165 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-18T11:22:37.165 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-18T11:22:37.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:22:37.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:22:37.165 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:22:37.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836505 2024-09-18T11:22:37.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836505 2024-09-18T11:22:37.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505' 2024-09-18T11:22:37.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:22:37.279 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:22:37.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672982 2024-09-18T11:22:37.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672982 2024-09-18T11:22:37.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672982' 2024-09-18T11:22:37.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:22:37.386 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:22:37.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509453 2024-09-18T11:22:37.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509453 2024-09-18T11:22:37.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672982 2-64424509453' 2024-09-18T11:22:37.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:22:37.492 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836505 2024-09-18T11:22:37.493 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:22:37.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:22:37.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836505 2024-09-18T11:22:37.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:22:37.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836505 2024-09-18T11:22:37.495 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836505 2024-09-18T11:22:37.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-18T11:22:37.496 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:22:37.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836502 -lt 21474836505 2024-09-18T11:22:37.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:22:38.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:22:38.910 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:22:39.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-18T11:22:39.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:22:39.263 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672982 2024-09-18T11:22:39.263 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:22:39.265 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:22:39.265 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672982 2024-09-18T11:22:39.265 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:22:39.265 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672982 2024-09-18T11:22:39.266 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672982 2024-09-18T11:22:39.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-18T11:22:39.266 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:22:39.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-18T11:22:39.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:22:39.645 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509453 2024-09-18T11:22:39.645 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:22:39.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:22:39.647 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509453 2024-09-18T11:22:39.647 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:22:39.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509453 2024-09-18T11:22:39.648 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509453 2024-09-18T11:22:39.648 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-18T11:22:39.648 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:22:40.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-18T11:22:40.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2024-09-18T11:22:40.005 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2024-09-18T11:22:40.005 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:22:40.005 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:22:40.005 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:22:40.005 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:22:40.005 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:22:40.005 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-osd.0.asok 2024-09-18T11:22:40.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2024-09-18T11:22:40.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41135/ceph-osd.0.asok dump_osd_network 2024-09-18T11:22:40.079 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:22:40.079 INFO:tasks.workunit.client.0.smithi157.stdout: "threshold": 1000, 2024-09-18T11:22:40.079 INFO:tasks.workunit.client.0.smithi157.stdout: "entries": [] 2024-09-18T11:22:40.079 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:22:40.087 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:22:40.087 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2024-09-18T11:22:40.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2024-09-18T11:22:40.090 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:22:40.090 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2024-09-18T11:22:40.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2024-09-18T11:22:40.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2024-09-18T11:22:40.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-18T11:22:40.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-18T11:22:40.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-18T11:22:40.095 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:22:40.095 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:22:40.095 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:22:40.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-mgr.x.asok 2024-09-18T11:22:40.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2024-09-18T11:22:40.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41135/ceph-mgr.x.asok dump_osd_network 2024-09-18T11:22:40.159 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:22:40.159 INFO:tasks.workunit.client.0.smithi157.stdout: "threshold": 1000, 2024-09-18T11:22:40.159 INFO:tasks.workunit.client.0.smithi157.stdout: "entries": [] 2024-09-18T11:22:40.159 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:22:40.166 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:22:40.166 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2024-09-18T11:22:40.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2024-09-18T11:22:40.169 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:22:40.169 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2024-09-18T11:22:40.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2024-09-18T11:22:40.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2024-09-18T11:22:40.172 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2024-09-18T11:22:40.173 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:22:40.173 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:22:40.173 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:22:40.173 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:22:40.173 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:22:40.173 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-osd.0.asok 2024-09-18T11:22:40.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2024-09-18T11:22:40.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41135/ceph-osd.0.asok dump_osd_network 0 2024-09-18T11:22:40.245 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:22:40.245 INFO:tasks.workunit.client.0.smithi157.stdout: "threshold": 0, 2024-09-18T11:22:40.245 INFO:tasks.workunit.client.0.smithi157.stdout: "entries": [ 2024-09-18T11:22:40.246 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:22:40.246 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:39 2024", 2024-09-18T11:22:40.246 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:22:40.246 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:22:40.246 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:22:40.246 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:22:40.246 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:22:40.246 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.253, 2024-09-18T11:22:40.246 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.309, 2024-09-18T11:22:40.246 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.331 2024-09-18T11:22:40.246 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.246 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:22:40.247 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.307, 2024-09-18T11:22:40.247 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.449, 2024-09-18T11:22:40.247 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.449 2024-09-18T11:22:40.247 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.247 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:22:40.247 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.307, 2024-09-18T11:22:40.247 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.449, 2024-09-18T11:22:40.247 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.449 2024-09-18T11:22:40.247 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.247 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.197 2024-09-18T11:22:40.247 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.247 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:22:40.247 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:35 2024", 2024-09-18T11:22:40.248 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:22:40.248 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:22:40.248 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:22:40.248 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:22:40.248 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:22:40.248 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.315, 2024-09-18T11:22:40.248 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.301, 2024-09-18T11:22:40.248 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.282 2024-09-18T11:22:40.248 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.248 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:22:40.248 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.427, 2024-09-18T11:22:40.249 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.427, 2024-09-18T11:22:40.249 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.427 2024-09-18T11:22:40.249 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.249 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:22:40.249 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.427, 2024-09-18T11:22:40.249 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.427, 2024-09-18T11:22:40.249 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.427 2024-09-18T11:22:40.249 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.249 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.178 2024-09-18T11:22:40.249 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.249 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:22:40.249 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:35 2024", 2024-09-18T11:22:40.249 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:22:40.249 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:22:40.250 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:22:40.250 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:22:40.250 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:22:40.250 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.300, 2024-09-18T11:22:40.250 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.309, 2024-09-18T11:22:40.250 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.268 2024-09-18T11:22:40.250 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.250 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:22:40.250 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.408, 2024-09-18T11:22:40.250 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.479, 2024-09-18T11:22:40.250 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.479 2024-09-18T11:22:40.250 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.251 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:22:40.251 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.408, 2024-09-18T11:22:40.251 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.479, 2024-09-18T11:22:40.251 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.479 2024-09-18T11:22:40.251 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.251 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.165 2024-09-18T11:22:40.251 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.251 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:22:40.251 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:39 2024", 2024-09-18T11:22:40.251 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:22:40.251 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:22:40.251 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:22:40.251 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:22:40.252 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:22:40.252 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.245, 2024-09-18T11:22:40.252 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.294, 2024-09-18T11:22:40.252 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.307 2024-09-18T11:22:40.252 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.252 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:22:40.252 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.294, 2024-09-18T11:22:40.252 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.461, 2024-09-18T11:22:40.252 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.461 2024-09-18T11:22:40.252 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.252 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:22:40.252 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.294, 2024-09-18T11:22:40.252 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.461, 2024-09-18T11:22:40.252 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.461 2024-09-18T11:22:40.253 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.253 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.206 2024-09-18T11:22:40.253 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:22:40.253 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:22:40.253 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:22:40.253 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:22:40.253 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2024-09-18T11:22:40.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2024-09-18T11:22:40.256 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:22:40.256 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2024-09-18T11:22:40.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2024-09-18T11:22:40.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2024-09-18T11:22:40.259 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-18T11:22:40.259 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-18T11:22:40.260 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-18T11:22:40.260 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:22:40.260 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:22:40.260 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:22:40.260 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-mgr.x.asok 2024-09-18T11:22:40.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2024-09-18T11:22:40.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41135/ceph-mgr.x.asok dump_osd_network 0 2024-09-18T11:22:40.324 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:22:40.324 INFO:tasks.workunit.client.0.smithi157.stdout: "threshold": 0, 2024-09-18T11:22:40.324 INFO:tasks.workunit.client.0.smithi157.stdout: "entries": [ 2024-09-18T11:22:40.324 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:22:40.324 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:34 2024", 2024-09-18T11:22:40.324 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:22:40.324 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 2, 2024-09-18T11:22:40.324 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:22:40.324 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:22:40.325 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:22:40.325 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.386, 2024-09-18T11:22:40.325 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.347, 2024-09-18T11:22:40.325 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.359 2024-09-18T11:22:40.325 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.325 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:22:40.325 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.295, 2024-09-18T11:22:40.325 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.259, 2024-09-18T11:22:40.325 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.259 2024-09-18T11:22:40.325 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.325 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:22:40.325 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.458, 2024-09-18T11:22:40.325 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.468, 2024-09-18T11:22:40.326 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.468 2024-09-18T11:22:40.326 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.326 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.258 2024-09-18T11:22:40.326 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.326 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:22:40.326 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:34 2024", 2024-09-18T11:22:40.326 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:22:40.326 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 2, 2024-09-18T11:22:40.326 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 0, 2024-09-18T11:22:40.326 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:22:40.326 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:22:40.326 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.369, 2024-09-18T11:22:40.326 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.332, 2024-09-18T11:22:40.326 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.346 2024-09-18T11:22:40.327 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.327 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:22:40.327 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.294, 2024-09-18T11:22:40.327 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.241, 2024-09-18T11:22:40.327 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.241 2024-09-18T11:22:40.327 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.327 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:22:40.327 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.436, 2024-09-18T11:22:40.327 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.450, 2024-09-18T11:22:40.327 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.450 2024-09-18T11:22:40.327 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.327 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.195 2024-09-18T11:22:40.327 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.327 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:22:40.328 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:34 2024", 2024-09-18T11:22:40.328 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:22:40.328 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 2, 2024-09-18T11:22:40.328 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:22:40.328 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:22:40.328 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:22:40.328 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.365, 2024-09-18T11:22:40.328 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.334, 2024-09-18T11:22:40.328 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.340 2024-09-18T11:22:40.328 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.328 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:22:40.328 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.315, 2024-09-18T11:22:40.328 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.242, 2024-09-18T11:22:40.329 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.242 2024-09-18T11:22:40.329 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.329 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:22:40.329 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.399, 2024-09-18T11:22:40.329 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.424, 2024-09-18T11:22:40.329 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.424 2024-09-18T11:22:40.329 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.329 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.216 2024-09-18T11:22:40.329 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.329 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:22:40.329 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:34 2024", 2024-09-18T11:22:40.329 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:22:40.329 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:22:40.330 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:22:40.330 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:22:40.330 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:22:40.330 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.353, 2024-09-18T11:22:40.330 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.327, 2024-09-18T11:22:40.330 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.337 2024-09-18T11:22:40.330 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.330 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:22:40.330 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.296, 2024-09-18T11:22:40.330 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.214, 2024-09-18T11:22:40.330 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.214 2024-09-18T11:22:40.330 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.330 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:22:40.331 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.449, 2024-09-18T11:22:40.331 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.449, 2024-09-18T11:22:40.331 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.449 2024-09-18T11:22:40.331 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.331 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.307 2024-09-18T11:22:40.331 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.331 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:22:40.331 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:34 2024", 2024-09-18T11:22:40.331 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:22:40.331 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 2, 2024-09-18T11:22:40.331 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 0, 2024-09-18T11:22:40.331 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:22:40.331 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:22:40.331 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.349, 2024-09-18T11:22:40.332 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.313, 2024-09-18T11:22:40.332 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.328 2024-09-18T11:22:40.332 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.332 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:22:40.332 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.283, 2024-09-18T11:22:40.332 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.214, 2024-09-18T11:22:40.332 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.214 2024-09-18T11:22:40.332 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.332 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:22:40.332 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.413, 2024-09-18T11:22:40.332 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.428, 2024-09-18T11:22:40.332 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.428 2024-09-18T11:22:40.332 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.332 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.229 2024-09-18T11:22:40.333 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.333 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:22:40.333 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:34 2024", 2024-09-18T11:22:40.333 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:22:40.333 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 1, 2024-09-18T11:22:40.333 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:22:40.333 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:22:40.333 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:22:40.333 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.318, 2024-09-18T11:22:40.333 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.346, 2024-09-18T11:22:40.333 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.321 2024-09-18T11:22:40.333 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.333 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:22:40.334 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.231, 2024-09-18T11:22:40.334 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.222, 2024-09-18T11:22:40.334 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.222 2024-09-18T11:22:40.334 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.334 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:22:40.334 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.457, 2024-09-18T11:22:40.334 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.490, 2024-09-18T11:22:40.334 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.490 2024-09-18T11:22:40.334 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.334 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.343 2024-09-18T11:22:40.334 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.334 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:22:40.334 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:34 2024", 2024-09-18T11:22:40.334 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:22:40.335 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:22:40.335 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:22:40.335 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:22:40.335 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:22:40.335 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.342, 2024-09-18T11:22:40.335 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.308, 2024-09-18T11:22:40.335 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.312 2024-09-18T11:22:40.335 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.335 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:22:40.335 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.279, 2024-09-18T11:22:40.335 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.200, 2024-09-18T11:22:40.335 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.200 2024-09-18T11:22:40.335 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.336 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:22:40.336 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.461, 2024-09-18T11:22:40.336 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.461, 2024-09-18T11:22:40.336 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.461 2024-09-18T11:22:40.336 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.336 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.294 2024-09-18T11:22:40.336 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.336 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:22:40.336 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:34 2024", 2024-09-18T11:22:40.336 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:22:40.336 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 1, 2024-09-18T11:22:40.336 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:22:40.336 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:22:40.336 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:22:40.337 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.298, 2024-09-18T11:22:40.337 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.335, 2024-09-18T11:22:40.337 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.328 2024-09-18T11:22:40.337 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.337 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:22:40.337 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.229, 2024-09-18T11:22:40.337 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.229, 2024-09-18T11:22:40.337 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.229 2024-09-18T11:22:40.337 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.337 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:22:40.337 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.426, 2024-09-18T11:22:40.337 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.493, 2024-09-18T11:22:40.337 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.493 2024-09-18T11:22:40.338 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.338 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.318 2024-09-18T11:22:40.338 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.338 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:22:40.338 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:35 2024", 2024-09-18T11:22:40.338 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:22:40.338 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 1, 2024-09-18T11:22:40.338 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 0, 2024-09-18T11:22:40.338 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:22:40.338 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:22:40.338 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.292, 2024-09-18T11:22:40.338 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.328, 2024-09-18T11:22:40.338 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.291 2024-09-18T11:22:40.338 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.339 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:22:40.339 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.232, 2024-09-18T11:22:40.339 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.232, 2024-09-18T11:22:40.339 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.203 2024-09-18T11:22:40.339 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.339 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:22:40.339 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.399, 2024-09-18T11:22:40.339 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.484, 2024-09-18T11:22:40.339 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.484 2024-09-18T11:22:40.339 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.339 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.299 2024-09-18T11:22:40.339 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.339 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:22:40.340 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:35 2024", 2024-09-18T11:22:40.340 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:22:40.340 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:22:40.340 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:22:40.340 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:22:40.340 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:22:40.340 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.315, 2024-09-18T11:22:40.340 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.301, 2024-09-18T11:22:40.340 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.282 2024-09-18T11:22:40.340 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.340 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:22:40.340 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.238, 2024-09-18T11:22:40.340 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.182, 2024-09-18T11:22:40.340 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.182 2024-09-18T11:22:40.341 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.341 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:22:40.341 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.427, 2024-09-18T11:22:40.341 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.427, 2024-09-18T11:22:40.341 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.427 2024-09-18T11:22:40.341 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.341 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.277 2024-09-18T11:22:40.341 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.341 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:22:40.341 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:35 2024", 2024-09-18T11:22:40.341 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:22:40.341 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 1, 2024-09-18T11:22:40.341 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 0, 2024-09-18T11:22:40.342 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:22:40.342 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:22:40.342 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.295, 2024-09-18T11:22:40.342 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.311, 2024-09-18T11:22:40.342 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.287 2024-09-18T11:22:40.342 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.342 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:22:40.342 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.240, 2024-09-18T11:22:40.342 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.207, 2024-09-18T11:22:40.342 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.192 2024-09-18T11:22:40.342 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.342 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:22:40.342 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.439, 2024-09-18T11:22:40.343 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.439, 2024-09-18T11:22:40.343 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.439 2024-09-18T11:22:40.343 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.343 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.283 2024-09-18T11:22:40.343 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.343 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:22:40.343 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:35 2024", 2024-09-18T11:22:40.343 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:22:40.343 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:22:40.343 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:22:40.343 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:22:40.343 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:22:40.343 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.300, 2024-09-18T11:22:40.344 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.309, 2024-09-18T11:22:40.344 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.268 2024-09-18T11:22:40.344 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.344 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:22:40.344 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.225, 2024-09-18T11:22:40.344 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.161, 2024-09-18T11:22:40.344 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.161 2024-09-18T11:22:40.344 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.344 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:22:40.344 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.408, 2024-09-18T11:22:40.344 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.479, 2024-09-18T11:22:40.344 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.479 2024-09-18T11:22:40.344 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:22:40.345 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.343 2024-09-18T11:22:40.345 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:22:40.345 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:22:40.345 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:22:40.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:22:40.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2024-09-18T11:22:40.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2024-09-18T11:22:40.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:22:40.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2024-09-18T11:22:40.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2024-09-18T11:22:40.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2024-09-18T11:23:00.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2024-09-18T11:23:00.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:23:00.339 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:23:00.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:23:00.708 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-18T11:23:00.709 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-18T11:23:00.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:23:00.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:23:00.709 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:23:00.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836511 2024-09-18T11:23:00.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836511 2024-09-18T11:23:00.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836511' 2024-09-18T11:23:00.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:23:00.815 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:23:00.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672989 2024-09-18T11:23:00.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672989 2024-09-18T11:23:00.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836511 1-42949672989' 2024-09-18T11:23:00.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:23:00.945 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:23:01.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509460 2024-09-18T11:23:01.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509460 2024-09-18T11:23:01.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836511 1-42949672989 2-64424509460' 2024-09-18T11:23:01.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:23:01.050 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836511 2024-09-18T11:23:01.050 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:23:01.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:23:01.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836511 2024-09-18T11:23:01.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:23:01.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836511 2024-09-18T11:23:01.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-09-18T11:23:01.053 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836511 2024-09-18T11:23:01.053 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:23:01.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836511 2024-09-18T11:23:01.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:23:02.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:23:02.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:23:02.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836512 -lt 21474836511 2024-09-18T11:23:02.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:23:02.805 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672989 2024-09-18T11:23:02.805 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:23:02.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:23:02.806 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672989 2024-09-18T11:23:02.806 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:23:02.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672989 2024-09-18T11:23:02.808 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672989 2024-09-18T11:23:02.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-09-18T11:23:02.808 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:23:03.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672989 -lt 42949672989 2024-09-18T11:23:03.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:23:03.167 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509460 2024-09-18T11:23:03.167 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:23:03.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:23:03.169 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509460 2024-09-18T11:23:03.169 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:23:03.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509460 2024-09-18T11:23:03.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-18T11:23:03.170 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509460 2024-09-18T11:23:03.170 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:23:03.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509460 -lt 64424509460 2024-09-18T11:23:03.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2024-09-18T11:23:03.518 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2024-09-18T11:23:03.518 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:23:03.519 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:23:03.519 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:23:03.519 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:03.519 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:03.519 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-osd.0.asok 2024-09-18T11:23:03.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2024-09-18T11:23:03.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41135/ceph-osd.0.asok dump_osd_network 2024-09-18T11:23:03.592 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:23:03.592 INFO:tasks.workunit.client.0.smithi157.stdout: "threshold": 1000, 2024-09-18T11:23:03.592 INFO:tasks.workunit.client.0.smithi157.stdout: "entries": [] 2024-09-18T11:23:03.592 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:23:03.599 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:03.599 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2024-09-18T11:23:03.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2024-09-18T11:23:03.602 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:03.602 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2024-09-18T11:23:03.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2024-09-18T11:23:03.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2024-09-18T11:23:03.606 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-18T11:23:03.606 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-18T11:23:03.606 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-18T11:23:03.606 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:23:03.607 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:03.607 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:03.607 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-mgr.x.asok 2024-09-18T11:23:03.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2024-09-18T11:23:03.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41135/ceph-mgr.x.asok dump_osd_network 2024-09-18T11:23:03.671 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:23:03.671 INFO:tasks.workunit.client.0.smithi157.stdout: "threshold": 1000, 2024-09-18T11:23:03.672 INFO:tasks.workunit.client.0.smithi157.stdout: "entries": [] 2024-09-18T11:23:03.672 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:23:03.679 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:03.679 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2024-09-18T11:23:03.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2024-09-18T11:23:03.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:03.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2024-09-18T11:23:03.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2024-09-18T11:23:03.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2024-09-18T11:23:03.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2024-09-18T11:23:03.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:23:03.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:23:03.686 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:23:03.686 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:03.686 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:03.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-osd.0.asok 2024-09-18T11:23:03.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2024-09-18T11:23:03.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41135/ceph-osd.0.asok dump_osd_network 0 2024-09-18T11:23:03.758 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:23:03.759 INFO:tasks.workunit.client.0.smithi157.stdout: "threshold": 0, 2024-09-18T11:23:03.759 INFO:tasks.workunit.client.0.smithi157.stdout: "entries": [ 2024-09-18T11:23:03.759 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:03.759 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:59 2024", 2024-09-18T11:23:03.759 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:03.759 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:23:03.759 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:23:03.759 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:23:03.759 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:03.759 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.292, 2024-09-18T11:23:03.759 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.301, 2024-09-18T11:23:03.759 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.323 2024-09-18T11:23:03.759 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.760 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:03.760 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.311, 2024-09-18T11:23:03.760 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.463, 2024-09-18T11:23:03.760 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.463 2024-09-18T11:23:03.760 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.760 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:03.760 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.311, 2024-09-18T11:23:03.760 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.463, 2024-09-18T11:23:03.760 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.463 2024-09-18T11:23:03.760 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.760 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.288 2024-09-18T11:23:03.760 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.760 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:03.761 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:59 2024", 2024-09-18T11:23:03.761 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:03.761 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:23:03.761 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:23:03.761 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:23:03.761 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:03.761 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.300, 2024-09-18T11:23:03.761 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.297, 2024-09-18T11:23:03.761 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.306 2024-09-18T11:23:03.761 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.761 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:03.762 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.332, 2024-09-18T11:23:03.762 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.472, 2024-09-18T11:23:03.762 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.472 2024-09-18T11:23:03.762 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.762 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:03.762 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.332, 2024-09-18T11:23:03.762 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.472, 2024-09-18T11:23:03.762 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.472 2024-09-18T11:23:03.762 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.762 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.276 2024-09-18T11:23:03.762 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.762 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:03.762 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:00 2024", 2024-09-18T11:23:03.763 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:03.763 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:23:03.763 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:23:03.763 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:23:03.763 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:03.763 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.276, 2024-09-18T11:23:03.763 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.281, 2024-09-18T11:23:03.763 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.280 2024-09-18T11:23:03.763 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.763 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:03.763 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.311, 2024-09-18T11:23:03.764 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.492, 2024-09-18T11:23:03.764 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.492 2024-09-18T11:23:03.764 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.764 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:03.764 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.311, 2024-09-18T11:23:03.764 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.492, 2024-09-18T11:23:03.764 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.492 2024-09-18T11:23:03.764 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.764 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.219 2024-09-18T11:23:03.764 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.764 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:03.764 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:00 2024", 2024-09-18T11:23:03.764 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:03.764 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:23:03.764 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:23:03.764 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:23:03.765 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:03.765 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.263, 2024-09-18T11:23:03.765 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.266, 2024-09-18T11:23:03.765 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.273 2024-09-18T11:23:03.765 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.765 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:03.765 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.345, 2024-09-18T11:23:03.765 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.434, 2024-09-18T11:23:03.765 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.479 2024-09-18T11:23:03.765 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.765 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:03.765 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.345, 2024-09-18T11:23:03.765 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.434, 2024-09-18T11:23:03.765 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.479 2024-09-18T11:23:03.766 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.766 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.253 2024-09-18T11:23:03.766 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:23:03.766 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:23:03.766 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:23:03.766 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:03.766 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2024-09-18T11:23:03.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2024-09-18T11:23:03.769 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:03.769 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2024-09-18T11:23:03.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2024-09-18T11:23:03.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2024-09-18T11:23:03.773 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-18T11:23:03.773 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-18T11:23:03.773 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-18T11:23:03.773 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:23:03.773 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:03.773 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:03.773 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-mgr.x.asok 2024-09-18T11:23:03.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2024-09-18T11:23:03.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41135/ceph-mgr.x.asok dump_osd_network 0 2024-09-18T11:23:03.838 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:23:03.838 INFO:tasks.workunit.client.0.smithi157.stdout: "threshold": 0, 2024-09-18T11:23:03.838 INFO:tasks.workunit.client.0.smithi157.stdout: "entries": [ 2024-09-18T11:23:03.838 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:03.838 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:59 2024", 2024-09-18T11:23:03.838 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:03.838 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 2, 2024-09-18T11:23:03.838 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:23:03.839 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:23:03.839 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:03.839 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.285, 2024-09-18T11:23:03.839 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.308, 2024-09-18T11:23:03.839 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.340 2024-09-18T11:23:03.839 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.839 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:03.839 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.233, 2024-09-18T11:23:03.839 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.203, 2024-09-18T11:23:03.839 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.203 2024-09-18T11:23:03.840 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.840 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:03.840 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.411, 2024-09-18T11:23:03.840 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.509, 2024-09-18T11:23:03.840 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.509 2024-09-18T11:23:03.840 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.840 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.288 2024-09-18T11:23:03.840 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.840 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:03.840 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:59 2024", 2024-09-18T11:23:03.840 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:03.840 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 1, 2024-09-18T11:23:03.840 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:23:03.841 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:23:03.841 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:03.841 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.290, 2024-09-18T11:23:03.841 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.331, 2024-09-18T11:23:03.841 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.328 2024-09-18T11:23:03.841 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.841 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:03.841 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.238, 2024-09-18T11:23:03.841 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.231, 2024-09-18T11:23:03.841 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.222 2024-09-18T11:23:03.841 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.841 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:03.841 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.331, 2024-09-18T11:23:03.842 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.471, 2024-09-18T11:23:03.842 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.490 2024-09-18T11:23:03.842 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.842 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.296 2024-09-18T11:23:03.842 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.842 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:03.842 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:59 2024", 2024-09-18T11:23:03.842 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:03.842 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 1, 2024-09-18T11:23:03.842 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:23:03.842 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:23:03.842 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:03.842 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.286, 2024-09-18T11:23:03.842 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.320, 2024-09-18T11:23:03.843 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.327 2024-09-18T11:23:03.843 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.843 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:03.843 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.260, 2024-09-18T11:23:03.843 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.204, 2024-09-18T11:23:03.843 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.204 2024-09-18T11:23:03.843 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.843 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:03.843 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.308, 2024-09-18T11:23:03.843 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.482, 2024-09-18T11:23:03.843 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.493 2024-09-18T11:23:03.843 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.843 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.281 2024-09-18T11:23:03.843 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.844 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:03.844 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:59 2024", 2024-09-18T11:23:03.844 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:03.844 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 2, 2024-09-18T11:23:03.844 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 0, 2024-09-18T11:23:03.844 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:23:03.844 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:03.844 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.270, 2024-09-18T11:23:03.844 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.285, 2024-09-18T11:23:03.844 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.323 2024-09-18T11:23:03.844 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.844 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:03.844 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.232, 2024-09-18T11:23:03.845 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.191, 2024-09-18T11:23:03.845 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.191 2024-09-18T11:23:03.845 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.845 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:03.845 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.338, 2024-09-18T11:23:03.845 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.483, 2024-09-18T11:23:03.845 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.483 2024-09-18T11:23:03.845 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.845 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.261 2024-09-18T11:23:03.845 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.845 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:03.845 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:59 2024", 2024-09-18T11:23:03.846 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:03.846 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:23:03.846 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:23:03.846 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:23:03.846 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:03.846 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.292, 2024-09-18T11:23:03.846 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.301, 2024-09-18T11:23:03.846 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.323 2024-09-18T11:23:03.846 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.846 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:03.846 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.272, 2024-09-18T11:23:03.846 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.197, 2024-09-18T11:23:03.846 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.197 2024-09-18T11:23:03.846 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.846 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:03.847 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.311, 2024-09-18T11:23:03.847 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.463, 2024-09-18T11:23:03.847 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.463 2024-09-18T11:23:03.847 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.847 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.218 2024-09-18T11:23:03.847 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.847 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:03.847 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:59 2024", 2024-09-18T11:23:03.847 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:03.847 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 2, 2024-09-18T11:23:03.847 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:23:03.847 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:23:03.847 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:03.847 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.294, 2024-09-18T11:23:03.848 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.291, 2024-09-18T11:23:03.848 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.322 2024-09-18T11:23:03.848 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.848 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:03.848 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.259, 2024-09-18T11:23:03.848 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.214, 2024-09-18T11:23:03.848 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.214 2024-09-18T11:23:03.848 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.848 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:03.848 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.354, 2024-09-18T11:23:03.848 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.458, 2024-09-18T11:23:03.848 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.458 2024-09-18T11:23:03.849 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.849 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.301 2024-09-18T11:23:03.849 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.849 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:03.849 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:59 2024", 2024-09-18T11:23:03.849 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:03.849 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 2, 2024-09-18T11:23:03.849 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 0, 2024-09-18T11:23:03.849 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:23:03.849 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:03.849 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.288, 2024-09-18T11:23:03.849 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.292, 2024-09-18T11:23:03.849 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.314 2024-09-18T11:23:03.849 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.850 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:03.850 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.218, 2024-09-18T11:23:03.850 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.213, 2024-09-18T11:23:03.850 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.213 2024-09-18T11:23:03.850 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.850 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:03.850 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.381, 2024-09-18T11:23:03.850 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.425, 2024-09-18T11:23:03.850 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.428 2024-09-18T11:23:03.850 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.850 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.233 2024-09-18T11:23:03.850 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.850 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:03.850 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:00 2024", 2024-09-18T11:23:03.850 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:03.851 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 1, 2024-09-18T11:23:03.851 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 0, 2024-09-18T11:23:03.851 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:23:03.851 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:03.851 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.261, 2024-09-18T11:23:03.851 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.314, 2024-09-18T11:23:03.851 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.300 2024-09-18T11:23:03.851 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.851 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:03.851 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.232, 2024-09-18T11:23:03.851 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.214, 2024-09-18T11:23:03.851 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.203 2024-09-18T11:23:03.851 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.851 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:03.852 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.291, 2024-09-18T11:23:03.852 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.500, 2024-09-18T11:23:03.852 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.500 2024-09-18T11:23:03.852 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.852 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.242 2024-09-18T11:23:03.852 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.852 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:03.852 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:22:59 2024", 2024-09-18T11:23:03.852 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:03.852 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:23:03.852 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:23:03.852 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:23:03.852 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:03.852 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.300, 2024-09-18T11:23:03.853 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.297, 2024-09-18T11:23:03.853 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.306 2024-09-18T11:23:03.853 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.853 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:03.853 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.263, 2024-09-18T11:23:03.853 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.206, 2024-09-18T11:23:03.853 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.200 2024-09-18T11:23:03.853 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.853 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:03.853 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.332, 2024-09-18T11:23:03.853 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.472, 2024-09-18T11:23:03.853 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.472 2024-09-18T11:23:03.853 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.853 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.233 2024-09-18T11:23:03.854 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.854 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:03.854 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:00 2024", 2024-09-18T11:23:03.854 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:03.854 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 1, 2024-09-18T11:23:03.854 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 0, 2024-09-18T11:23:03.854 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:23:03.854 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:03.854 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.246, 2024-09-18T11:23:03.854 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.288, 2024-09-18T11:23:03.854 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.285 2024-09-18T11:23:03.854 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.854 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:03.854 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.203, 2024-09-18T11:23:03.855 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.177, 2024-09-18T11:23:03.855 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.177 2024-09-18T11:23:03.855 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.855 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:03.855 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.271, 2024-09-18T11:23:03.855 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.437, 2024-09-18T11:23:03.855 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.439 2024-09-18T11:23:03.855 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.855 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.258 2024-09-18T11:23:03.855 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.855 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:03.855 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:00 2024", 2024-09-18T11:23:03.855 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:03.856 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:23:03.856 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:23:03.856 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:23:03.856 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:03.856 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.276, 2024-09-18T11:23:03.856 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.281, 2024-09-18T11:23:03.856 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.280 2024-09-18T11:23:03.856 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.856 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:03.856 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.248, 2024-09-18T11:23:03.856 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.178, 2024-09-18T11:23:03.856 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.178 2024-09-18T11:23:03.857 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.857 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:03.857 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.311, 2024-09-18T11:23:03.857 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.492, 2024-09-18T11:23:03.857 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.492 2024-09-18T11:23:03.857 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.857 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.248 2024-09-18T11:23:03.857 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.857 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:03.857 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:00 2024", 2024-09-18T11:23:03.857 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:03.857 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:23:03.857 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:23:03.857 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:23:03.858 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:03.858 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.263, 2024-09-18T11:23:03.858 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.266, 2024-09-18T11:23:03.858 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.273 2024-09-18T11:23:03.858 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.858 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:03.858 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.187, 2024-09-18T11:23:03.858 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.165, 2024-09-18T11:23:03.858 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.161 2024-09-18T11:23:03.858 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.858 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:03.858 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.345, 2024-09-18T11:23:03.858 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.434, 2024-09-18T11:23:03.859 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.479 2024-09-18T11:23:03.859 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:03.859 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.187 2024-09-18T11:23:03.859 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:23:03.859 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:23:03.859 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:23:03.860 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:03.860 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2024-09-18T11:23:03.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2024-09-18T11:23:03.860 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:03.860 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2024-09-18T11:23:03.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2024-09-18T11:23:03.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2024-09-18T11:23:53.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2024-09-18T11:23:53.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:23:53.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:23:54.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:23:54.234 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-18T11:23:54.234 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-18T11:23:54.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:23:54.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:23:54.235 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:23:54.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836524 2024-09-18T11:23:54.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836524 2024-09-18T11:23:54.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524' 2024-09-18T11:23:54.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:23:54.341 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:23:54.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673001 2024-09-18T11:23:54.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673001 2024-09-18T11:23:54.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949673001' 2024-09-18T11:23:54.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:23:54.450 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:23:54.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509473 2024-09-18T11:23:54.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509473 2024-09-18T11:23:54.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949673001 2-64424509473' 2024-09-18T11:23:54.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:23:54.572 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836524 2024-09-18T11:23:54.573 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:23:54.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:23:54.574 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836524 2024-09-18T11:23:54.574 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:23:54.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836524 2024-09-18T11:23:54.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-18T11:23:54.575 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836524 2024-09-18T11:23:54.575 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:23:54.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836522 -lt 21474836524 2024-09-18T11:23:54.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:23:55.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:23:55.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:23:56.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-18T11:23:56.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:23:56.329 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673001 2024-09-18T11:23:56.329 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:23:56.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:23:56.331 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673001 2024-09-18T11:23:56.331 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:23:56.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673001 2024-09-18T11:23:56.332 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949673001 2024-09-18T11:23:56.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2024-09-18T11:23:56.332 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:23:56.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673002 -lt 42949673001 2024-09-18T11:23:56.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:23:56.683 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509473 2024-09-18T11:23:56.683 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:23:56.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:23:56.684 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509473 2024-09-18T11:23:56.684 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:23:56.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509473 2024-09-18T11:23:56.685 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509473 2024-09-18T11:23:56.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2024-09-18T11:23:56.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:23:57.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509473 -lt 64424509473 2024-09-18T11:23:57.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2024-09-18T11:23:57.046 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2024-09-18T11:23:57.046 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:23:57.046 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:23:57.046 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:23:57.047 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:57.047 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:57.047 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-osd.0.asok 2024-09-18T11:23:57.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2024-09-18T11:23:57.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41135/ceph-osd.0.asok dump_osd_network 2024-09-18T11:23:57.119 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:23:57.119 INFO:tasks.workunit.client.0.smithi157.stdout: "threshold": 1000, 2024-09-18T11:23:57.119 INFO:tasks.workunit.client.0.smithi157.stdout: "entries": [] 2024-09-18T11:23:57.119 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:23:57.126 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:57.126 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2024-09-18T11:23:57.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2024-09-18T11:23:57.129 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:57.129 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2024-09-18T11:23:57.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2024-09-18T11:23:57.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2024-09-18T11:23:57.133 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-18T11:23:57.133 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-18T11:23:57.133 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-18T11:23:57.133 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:23:57.133 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:57.133 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:57.133 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-mgr.x.asok 2024-09-18T11:23:57.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2024-09-18T11:23:57.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41135/ceph-mgr.x.asok dump_osd_network 2024-09-18T11:23:57.197 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:23:57.198 INFO:tasks.workunit.client.0.smithi157.stdout: "threshold": 1000, 2024-09-18T11:23:57.198 INFO:tasks.workunit.client.0.smithi157.stdout: "entries": [] 2024-09-18T11:23:57.198 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:23:57.205 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:57.205 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2024-09-18T11:23:57.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2024-09-18T11:23:57.208 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:57.208 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2024-09-18T11:23:57.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2024-09-18T11:23:57.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2024-09-18T11:23:57.211 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2024-09-18T11:23:57.211 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:23:57.212 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:23:57.212 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:23:57.212 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:57.212 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:57.212 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-osd.0.asok 2024-09-18T11:23:57.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2024-09-18T11:23:57.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41135/ceph-osd.0.asok dump_osd_network 0 2024-09-18T11:23:57.285 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:23:57.286 INFO:tasks.workunit.client.0.smithi157.stdout: "threshold": 0, 2024-09-18T11:23:57.286 INFO:tasks.workunit.client.0.smithi157.stdout: "entries": [ 2024-09-18T11:23:57.286 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:57.286 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:54 2024", 2024-09-18T11:23:57.286 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:57.286 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:23:57.286 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:23:57.286 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:23:57.286 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:57.286 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.314, 2024-09-18T11:23:57.286 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.315, 2024-09-18T11:23:57.286 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.324 2024-09-18T11:23:57.287 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.287 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:57.287 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.425, 2024-09-18T11:23:57.287 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.458, 2024-09-18T11:23:57.287 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.500 2024-09-18T11:23:57.287 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.287 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:57.287 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.425, 2024-09-18T11:23:57.287 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.458, 2024-09-18T11:23:57.287 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.500 2024-09-18T11:23:57.287 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.287 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.270 2024-09-18T11:23:57.287 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.288 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:57.288 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:54 2024", 2024-09-18T11:23:57.288 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:57.288 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:23:57.288 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:23:57.288 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:23:57.288 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:57.288 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.309, 2024-09-18T11:23:57.288 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.304, 2024-09-18T11:23:57.288 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.316 2024-09-18T11:23:57.288 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.288 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:57.289 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.435, 2024-09-18T11:23:57.289 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.443, 2024-09-18T11:23:57.289 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.472 2024-09-18T11:23:57.289 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.289 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:57.289 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.435, 2024-09-18T11:23:57.289 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.443, 2024-09-18T11:23:57.289 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.472 2024-09-18T11:23:57.289 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.289 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.287 2024-09-18T11:23:57.289 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.289 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:57.289 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:55 2024", 2024-09-18T11:23:57.289 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:57.290 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:23:57.290 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:23:57.290 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:23:57.290 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:57.290 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.270, 2024-09-18T11:23:57.290 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.303, 2024-09-18T11:23:57.290 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.308 2024-09-18T11:23:57.290 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.290 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:57.290 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.311, 2024-09-18T11:23:57.290 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.419, 2024-09-18T11:23:57.290 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.492 2024-09-18T11:23:57.290 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.290 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:57.291 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.311, 2024-09-18T11:23:57.291 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.419, 2024-09-18T11:23:57.291 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.492 2024-09-18T11:23:57.291 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.291 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.312 2024-09-18T11:23:57.291 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.291 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:57.291 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:55 2024", 2024-09-18T11:23:57.291 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:57.291 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:23:57.291 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:23:57.291 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:23:57.291 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:57.292 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.268, 2024-09-18T11:23:57.292 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.293, 2024-09-18T11:23:57.292 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.294 2024-09-18T11:23:57.292 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.292 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:57.292 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.301, 2024-09-18T11:23:57.292 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.406, 2024-09-18T11:23:57.292 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.491 2024-09-18T11:23:57.292 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.292 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:57.292 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.301, 2024-09-18T11:23:57.292 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.406, 2024-09-18T11:23:57.292 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.491 2024-09-18T11:23:57.293 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.293 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.302 2024-09-18T11:23:57.293 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:23:57.293 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:23:57.293 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:23:57.293 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:57.293 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2024-09-18T11:23:57.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2024-09-18T11:23:57.296 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:57.296 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2024-09-18T11:23:57.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2024-09-18T11:23:57.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2024-09-18T11:23:57.300 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-18T11:23:57.300 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-18T11:23:57.300 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-18T11:23:57.300 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:23:57.300 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:57.300 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:57.300 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-mgr.x.asok 2024-09-18T11:23:57.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2024-09-18T11:23:57.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41135/ceph-mgr.x.asok dump_osd_network 0 2024-09-18T11:23:57.367 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:23:57.367 INFO:tasks.workunit.client.0.smithi157.stdout: "threshold": 0, 2024-09-18T11:23:57.367 INFO:tasks.workunit.client.0.smithi157.stdout: "entries": [ 2024-09-18T11:23:57.367 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:57.367 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:50 2024", 2024-09-18T11:23:57.367 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:57.367 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:23:57.367 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:23:57.368 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:23:57.368 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:57.368 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.334, 2024-09-18T11:23:57.368 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.295, 2024-09-18T11:23:57.368 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.293 2024-09-18T11:23:57.368 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.368 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:57.368 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.270, 2024-09-18T11:23:57.368 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.206, 2024-09-18T11:23:57.368 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.165 2024-09-18T11:23:57.368 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.369 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:57.369 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.380, 2024-09-18T11:23:57.369 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.406, 2024-09-18T11:23:57.369 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.491 2024-09-18T11:23:57.369 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.369 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.281 2024-09-18T11:23:57.369 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.369 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:57.369 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:54 2024", 2024-09-18T11:23:57.369 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:57.369 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 2, 2024-09-18T11:23:57.369 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:23:57.370 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:23:57.370 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:57.370 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.299, 2024-09-18T11:23:57.370 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.318, 2024-09-18T11:23:57.370 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.326 2024-09-18T11:23:57.370 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.370 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:57.370 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.267, 2024-09-18T11:23:57.370 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.248, 2024-09-18T11:23:57.370 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.233 2024-09-18T11:23:57.370 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.370 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:57.370 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.359, 2024-09-18T11:23:57.371 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.433, 2024-09-18T11:23:57.371 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.509 2024-09-18T11:23:57.371 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.371 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.276 2024-09-18T11:23:57.371 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.371 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:57.371 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:54 2024", 2024-09-18T11:23:57.371 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:57.371 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 1, 2024-09-18T11:23:57.371 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:23:57.372 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:23:57.372 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:57.372 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.285, 2024-09-18T11:23:57.372 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.301, 2024-09-18T11:23:57.372 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.326 2024-09-18T11:23:57.372 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.372 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:57.372 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.214, 2024-09-18T11:23:57.372 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.213, 2024-09-18T11:23:57.372 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.213 2024-09-18T11:23:57.373 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.373 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:57.373 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.420, 2024-09-18T11:23:57.373 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.464, 2024-09-18T11:23:57.373 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.478 2024-09-18T11:23:57.373 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.373 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.181 2024-09-18T11:23:57.373 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.373 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:57.373 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:49 2024", 2024-09-18T11:23:57.373 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:57.374 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:23:57.374 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:23:57.374 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:23:57.374 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:57.374 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.305, 2024-09-18T11:23:57.374 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.326, 2024-09-18T11:23:57.374 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.320 2024-09-18T11:23:57.374 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.374 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:57.374 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.221, 2024-09-18T11:23:57.374 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.221, 2024-09-18T11:23:57.374 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.197 2024-09-18T11:23:57.375 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.375 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:57.375 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.396, 2024-09-18T11:23:57.375 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.465, 2024-09-18T11:23:57.375 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.500 2024-09-18T11:23:57.375 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.375 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.320 2024-09-18T11:23:57.375 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.375 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:57.375 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:50 2024", 2024-09-18T11:23:57.375 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:57.375 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:23:57.375 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:23:57.375 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:23:57.376 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:57.376 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.321, 2024-09-18T11:23:57.376 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.311, 2024-09-18T11:23:57.376 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.307 2024-09-18T11:23:57.376 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.376 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:57.376 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.194, 2024-09-18T11:23:57.376 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.194, 2024-09-18T11:23:57.376 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.178 2024-09-18T11:23:57.376 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.376 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:57.377 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.398, 2024-09-18T11:23:57.377 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.428, 2024-09-18T11:23:57.377 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.492 2024-09-18T11:23:57.377 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.377 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.298 2024-09-18T11:23:57.377 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.377 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:57.377 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:54 2024", 2024-09-18T11:23:57.377 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:57.377 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 2, 2024-09-18T11:23:57.377 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 1, 2024-09-18T11:23:57.377 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:23:57.377 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:57.378 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.285, 2024-09-18T11:23:57.378 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.311, 2024-09-18T11:23:57.378 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.314 2024-09-18T11:23:57.378 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.378 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:57.378 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.251, 2024-09-18T11:23:57.378 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.210, 2024-09-18T11:23:57.378 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.210 2024-09-18T11:23:57.378 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.378 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:57.378 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.329, 2024-09-18T11:23:57.379 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.429, 2024-09-18T11:23:57.379 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.461 2024-09-18T11:23:57.379 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.379 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.261 2024-09-18T11:23:57.379 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.379 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:57.379 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:54 2024", 2024-09-18T11:23:57.379 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:57.379 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 1, 2024-09-18T11:23:57.379 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:23:57.379 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:23:57.379 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:57.379 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.292, 2024-09-18T11:23:57.380 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.293, 2024-09-18T11:23:57.380 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.314 2024-09-18T11:23:57.380 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.380 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:57.380 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.201, 2024-09-18T11:23:57.380 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.201, 2024-09-18T11:23:57.380 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.201 2024-09-18T11:23:57.380 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.380 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:57.380 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.441, 2024-09-18T11:23:57.380 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.448, 2024-09-18T11:23:57.380 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.482 2024-09-18T11:23:57.380 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.381 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.196 2024-09-18T11:23:57.381 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.381 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:57.381 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:49 2024", 2024-09-18T11:23:57.381 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:57.381 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 0, 2024-09-18T11:23:57.381 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 2, 2024-09-18T11:23:57.381 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:23:57.381 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:57.381 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.302, 2024-09-18T11:23:57.381 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.311, 2024-09-18T11:23:57.381 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.311 2024-09-18T11:23:57.381 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.382 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:57.382 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.245, 2024-09-18T11:23:57.382 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.222, 2024-09-18T11:23:57.382 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.206 2024-09-18T11:23:57.382 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.382 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:57.382 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.368, 2024-09-18T11:23:57.382 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.451, 2024-09-18T11:23:57.382 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.472 2024-09-18T11:23:57.382 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.382 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.332 2024-09-18T11:23:57.382 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.382 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:57.382 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:55 2024", 2024-09-18T11:23:57.383 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:57.383 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 1, 2024-09-18T11:23:57.383 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 0, 2024-09-18T11:23:57.383 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:23:57.383 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:57.383 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.291, 2024-09-18T11:23:57.383 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.301, 2024-09-18T11:23:57.383 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.309 2024-09-18T11:23:57.383 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.383 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:57.383 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.217, 2024-09-18T11:23:57.383 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.217, 2024-09-18T11:23:57.383 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.209 2024-09-18T11:23:57.383 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.383 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:57.384 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.473, 2024-09-18T11:23:57.384 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.473, 2024-09-18T11:23:57.384 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.500 2024-09-18T11:23:57.384 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.384 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.217 2024-09-18T11:23:57.384 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.384 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:57.384 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:54 2024", 2024-09-18T11:23:57.384 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:57.384 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 2, 2024-09-18T11:23:57.384 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 0, 2024-09-18T11:23:57.384 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:23:57.384 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:57.385 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.291, 2024-09-18T11:23:57.385 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.292, 2024-09-18T11:23:57.385 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.301 2024-09-18T11:23:57.385 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.385 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:57.385 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.231, 2024-09-18T11:23:57.385 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.222, 2024-09-18T11:23:57.385 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.218 2024-09-18T11:23:57.385 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.385 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:57.386 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.353, 2024-09-18T11:23:57.386 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.441, 2024-09-18T11:23:57.386 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.520 2024-09-18T11:23:57.386 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.386 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.231 2024-09-18T11:23:57.386 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.386 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:57.386 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:54 2024", 2024-09-18T11:23:57.386 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:57.386 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 2, 2024-09-18T11:23:57.386 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 0, 2024-09-18T11:23:57.386 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "front", 2024-09-18T11:23:57.386 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:57.387 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.284, 2024-09-18T11:23:57.387 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.287, 2024-09-18T11:23:57.387 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.300 2024-09-18T11:23:57.387 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.387 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:57.387 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.243, 2024-09-18T11:23:57.387 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.205, 2024-09-18T11:23:57.387 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.205 2024-09-18T11:23:57.387 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.387 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:57.387 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.338, 2024-09-18T11:23:57.387 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.377, 2024-09-18T11:23:57.387 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.483 2024-09-18T11:23:57.387 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.388 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.243 2024-09-18T11:23:57.388 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.388 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:23:57.388 INFO:tasks.workunit.client.0.smithi157.stdout: "last update": "Wed Sep 18 11:23:55 2024", 2024-09-18T11:23:57.388 INFO:tasks.workunit.client.0.smithi157.stdout: "stale": false, 2024-09-18T11:23:57.388 INFO:tasks.workunit.client.0.smithi157.stdout: "from osd": 1, 2024-09-18T11:23:57.388 INFO:tasks.workunit.client.0.smithi157.stdout: "to osd": 0, 2024-09-18T11:23:57.388 INFO:tasks.workunit.client.0.smithi157.stdout: "interface": "back", 2024-09-18T11:23:57.388 INFO:tasks.workunit.client.0.smithi157.stdout: "average": { 2024-09-18T11:23:57.388 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.255, 2024-09-18T11:23:57.388 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.266, 2024-09-18T11:23:57.389 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.286 2024-09-18T11:23:57.389 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.389 INFO:tasks.workunit.client.0.smithi157.stdout: "min": { 2024-09-18T11:23:57.389 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.206, 2024-09-18T11:23:57.389 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.198, 2024-09-18T11:23:57.389 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.198 2024-09-18T11:23:57.389 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.389 INFO:tasks.workunit.client.0.smithi157.stdout: "max": { 2024-09-18T11:23:57.389 INFO:tasks.workunit.client.0.smithi157.stdout: "1min": 0.393, 2024-09-18T11:23:57.389 INFO:tasks.workunit.client.0.smithi157.stdout: "5min": 0.422, 2024-09-18T11:23:57.390 INFO:tasks.workunit.client.0.smithi157.stdout: "15min": 0.443 2024-09-18T11:23:57.390 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:23:57.390 INFO:tasks.workunit.client.0.smithi157.stdout: "last": 0.228 2024-09-18T11:23:57.390 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:23:57.390 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:23:57.390 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:23:57.392 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:57.392 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2024-09-18T11:23:57.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2024-09-18T11:23:57.392 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:57.392 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2024-09-18T11:23:57.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2024-09-18T11:23:57.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2024-09-18T11:23:57.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-18T11:23:57.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-18T11:23:57.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-18T11:23:57.393 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:23:57.393 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:57.393 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:57.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-mgr.x.asok 2024-09-18T11:23:57.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2024-09-18T11:23:57.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41135/ceph-mgr.x.asok dump_osd_network 99 2024-09-18T11:23:57.445 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:23:57.445 INFO:tasks.workunit.client.0.smithi157.stdout: "threshold": 99, 2024-09-18T11:23:57.445 INFO:tasks.workunit.client.0.smithi157.stdout: "entries": [] 2024-09-18T11:23:57.445 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:23:57.452 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:57.453 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2024-09-18T11:23:57.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2024-09-18T11:23:57.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2024-09-18T11:23:57.456 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2024-09-18T11:23:57.456 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:23:57.456 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:23:57.457 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:23:57.457 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:57.457 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:57.457 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-osd.0.asok 2024-09-18T11:23:57.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2024-09-18T11:23:57.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41135/ceph-osd.0.asok dump_osd_network 98 2024-09-18T11:23:57.528 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:23:57.529 INFO:tasks.workunit.client.0.smithi157.stdout: "threshold": 98, 2024-09-18T11:23:57.529 INFO:tasks.workunit.client.0.smithi157.stdout: "entries": [] 2024-09-18T11:23:57.529 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:23:57.536 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2024-09-18T11:23:57.536 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2024-09-18T11:23:57.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2024-09-18T11:23:57.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2024-09-18T11:23:57.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-09-18T11:23:57.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-18T11:23:57.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:23:57.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-18T11:23:57.540 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:23:57.540 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:23:57.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:23:57.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:23:57.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:23:57.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:23:57.647 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:23:57.648 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:23:57.648 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:23:57.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:23:57.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:23:57.650 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:23:57.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:23:57.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:23:57.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:23:57.651 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:23:57.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:23:57.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:23:57.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-18T11:23:57.672 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:23:57.672 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:57.672 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:57.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.41135 2024-09-18T11:23:57.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:23:57.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:23:57.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-09-18T11:23:57.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-09-18T11:23:57.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-09-18T11:23:57.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-09-18T11:23:57.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-18T11:23:57.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:23:57.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-18T11:23:57.675 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:23:57.675 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:23:57.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:23:57.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:23:57.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:23:57.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:23:57.675 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:23:57.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:23:57.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:23:57.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:23:57.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:23:57.677 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:23:57.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:23:57.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:23:57.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:23:57.679 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:23:57.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:23:57.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:23:57.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-18T11:23:57.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:23:57.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:57.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:57.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.41135 2024-09-18T11:23:57.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:23:57.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:23:57.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-09-18T11:23:57.684 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:23:57.684 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:57.684 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:57.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.41135 2024-09-18T11:23:57.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:23:57.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:23:57.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:23:57.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-09-18T11:23:57.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2024-09-18T11:23:57.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2024-09-18T11:23:57.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2024-09-18T11:23:57.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2024-09-18T11:23:57.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2024-09-18T11:23:57.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2024-09-18T11:23:57.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-09-18T11:23:57.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:23:57.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:23:57.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:23:57.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-09-18T11:23:57.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-09-18T11:23:58.093 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:23:58.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:23:58.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:23:58.094 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:23:58.094 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:58.094 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:58.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41135/$cluster-$name.asok' 2024-09-18T11:23:58.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:23:58.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:23:58.330 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:23:58.330 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:23:58.330 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:23:58.330 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:23:58.330 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:23:58.330 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:23:58.330 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:23:58.330 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:23:58.331 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:23:58.331 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:58.331 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:58.331 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-mon.a.asok 2024-09-18T11:23:58.331 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:23:58.331 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.41135/ceph-mon.a.asok config get fsid 2024-09-18T11:23:58.409 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:23:58.409 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:23:58.409 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:23:58.409 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:23:58.409 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:23:58.409 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:23:58.409 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:23:58.409 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:23:58.409 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:23:58.409 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:58.410 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:58.410 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41135/ceph-mon.a.asok 2024-09-18T11:23:58.410 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:23:58.410 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.41135/ceph-mon.a.asok config get mon_host 2024-09-18T11:23:58.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2024-09-18T11:23:58.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-09-18T11:23:58.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:23:58.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:23:58.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:23:58.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-09-18T11:23:58.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:23:59.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:23:59.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:23:59.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:23:59.324 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:23:59.324 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:59.324 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:59.324 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41135/$cluster-$name.asok' 2024-09-18T11:23:59.324 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:23:59.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:23:59.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2024-09-18T11:23:59.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-18T11:23:59.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:23:59.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:23:59.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:23:59.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-09-18T11:23:59.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-18T11:23:59.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:23:59.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:23:59.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:23:59.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-09-18T11:23:59.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-09-18T11:23:59.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:23:59.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-18T11:23:59.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-18T11:23:59.353 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:23:59.353 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:23:59.353 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:23:59.353 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:23:59.353 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:59.353 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:23:59.353 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41135/$cluster-$name.asok' 2024-09-18T11:23:59.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' 2024-09-18T11:23:59.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:23:59.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:23:59.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:23:59.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-18T11:23:59.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-18T11:23:59.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:23:59.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:23:59.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:23:59.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:23:59.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:23:59.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:23:59.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-09-18T11:23:59.354 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:23:59.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bc025a2e-301d-472f-8c32-117e6c912916 2024-09-18T11:23:59.355 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 bc025a2e-301d-472f-8c32-117e6c912916 2024-09-18T11:23:59.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bc025a2e-301d-472f-8c32-117e6c912916' 2024-09-18T11:23:59.355 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:23:59.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBPuOpmlCvUFRAAPEemxjpJsZYfsb9yvt3B/Q== 2024-09-18T11:23:59.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBPuOpmlCvUFRAAPEemxjpJsZYfsb9yvt3B/Q=="}' 2024-09-18T11:23:59.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bc025a2e-301d-472f-8c32-117e6c912916 -i td/network-ping/0/new.json 2024-09-18T11:23:59.666 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:23:59.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-09-18T11:23:59.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBPuOpmlCvUFRAAPEemxjpJsZYfsb9yvt3B/Q== --osd-uuid bc025a2e-301d-472f-8c32-117e6c912916 2024-09-18T11:23:59.703 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:23:59.701+0000 7f5da034c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:23:59.713 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:23:59.711+0000 7f5da034c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:23:59.714 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:23:59.713+0000 7f5da034c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:23:59.714 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:23:59.713+0000 7f5da034c440 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-18T11:23:59.714 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:23:59.713+0000 7f5da034c440 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-18T11:23:59.714 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:23:59.713+0000 7f5da034c440 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-18T11:23:59.744 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:23:59.743+0000 7f5da034c440 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-09-18T11:24:05.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-09-18T11:24:05.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:24:05.072 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:24:05.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:24:05.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:24:05.551 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:24:05.552 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:24:05.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:24:05.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:24:05.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:24:05.553 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:24:05.576 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:05.574+0000 7f0c4e729440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:24:05.578 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:05.576+0000 7f0c4e729440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:24:05.579 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:05.577+0000 7f0c4e729440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:24:05.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:24:05.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:24:05.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:24:05.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:24:05.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:24:05.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:05.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:24:05.910 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:24:05.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:05.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:06.096 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:06.095+0000 7f0c4e729440 -1 Falling back to public interface 2024-09-18T11:24:06.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:07.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:07.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:07.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:24:07.259 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:24:07.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:07.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:07.615 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:08.577 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:08.575+0000 7f0c4e729440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:24:08.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:08.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:08.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:24:08.617 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:24:08.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:08.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:09.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:10.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:10.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:10.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:24:10.156 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:24:10.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:10.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:10.548 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:11.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:11.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:11.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:24:11.550 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:24:11.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:11.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:11.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:12.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:12.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:12.909 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:24:12.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:24:12.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:12.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:13.381 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:14.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:14.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:14.383 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:24:14.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:24:14.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:14.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:14.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:15.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:15.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:15.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:24:15.769 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:24:15.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:15.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:16.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:17.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:17.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:17.284 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:24:17.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:24:17.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:17.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:17.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:18.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:18.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:18.663 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:24:18.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:24:18.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:18.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:19.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:20.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:20.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:20.043 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:24:20.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:24:20.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:20.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:20.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:21.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:21.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:21.848 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:24:21.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:24:21.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:21.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:22.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:23.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:23.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:23.323 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:24:23.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:24:23.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:23.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:24.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:25.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:25.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:25.154 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:24:25.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:24:25.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:25.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:25.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:26.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:26.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:26.939 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:24:26.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:24:26.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:26.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:27.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:28.368 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:28.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:28.369 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:24:28.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:24:28.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:28.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:29.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:30.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:30.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:30.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:24:30.146 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:24:30.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:30.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:24:30.722 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/48574,v1:127.0.0.1:6803/48574] [v2:127.0.0.1:6804/48574,v1:127.0.0.1:6805/48574] exists,up bc025a2e-301d-472f-8c32-117e6c912916 2024-09-18T11:24:30.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:24:30.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:24:30.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:24:30.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2024-09-18T11:24:30.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-18T11:24:30.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:24:30.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:24:30.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:24:30.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-09-18T11:24:30.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-18T11:24:30.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:24:30.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:24:30.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:24:30.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-09-18T11:24:30.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-09-18T11:24:30.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:24:30.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-18T11:24:30.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-18T11:24:30.728 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:24:30.728 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:24:30.728 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:24:30.728 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:24:30.728 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:24:30.728 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:24:30.728 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41135/$cluster-$name.asok' 2024-09-18T11:24:30.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' 2024-09-18T11:24:30.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:24:30.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:24:30.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:24:30.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-18T11:24:30.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-18T11:24:30.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:24:30.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:24:30.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:24:30.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:24:30.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:24:30.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:24:30.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-09-18T11:24:30.730 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:24:30.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ba24956b-1eb9-4d1f-96f9-938759a0b7bd 2024-09-18T11:24:30.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ba24956b-1eb9-4d1f-96f9-938759a0b7bd' 2024-09-18T11:24:30.730 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 ba24956b-1eb9-4d1f-96f9-938759a0b7bd 2024-09-18T11:24:30.730 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:24:30.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBuuOpmBrMWLBAAY9dC411R/SuUlfdrbmFRkw== 2024-09-18T11:24:30.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBuuOpmBrMWLBAAY9dC411R/SuUlfdrbmFRkw=="}' 2024-09-18T11:24:30.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ba24956b-1eb9-4d1f-96f9-938759a0b7bd -i td/network-ping/1/new.json 2024-09-18T11:24:31.237 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:24:31.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-09-18T11:24:31.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBuuOpmBrMWLBAAY9dC411R/SuUlfdrbmFRkw== --osd-uuid ba24956b-1eb9-4d1f-96f9-938759a0b7bd 2024-09-18T11:24:31.274 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:31.272+0000 7f84f472a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:24:31.294 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:31.292+0000 7f84f472a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:24:31.295 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:31.293+0000 7f84f472a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:24:31.295 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:31.293+0000 7f84f472a440 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-18T11:24:31.296 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:31.293+0000 7f84f472a440 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-18T11:24:31.296 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:31.293+0000 7f84f472a440 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-18T11:24:31.378 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:31.376+0000 7f84f472a440 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-09-18T11:24:37.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-09-18T11:24:37.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:24:37.524 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T11:24:37.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:24:37.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:24:38.009 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T11:24:38.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:24:38.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:24:38.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:24:38.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:24:38.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:24:38.034 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:38.032+0000 7f8619e16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:24:38.036 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:38.034+0000 7f8619e16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:24:38.037 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:38.035+0000 7f8619e16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:24:38.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:24:38.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:24:38.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:24:38.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:24:38.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:24:38.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:38.363 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:24:38.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:24:38.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:38.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:24:38.568 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:38.566+0000 7f8619e16440 -1 Falling back to public interface 2024-09-18T11:24:38.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:39.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:39.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:39.718 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:24:39.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:24:39.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:39.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:24:40.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:41.104 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:24:41.103+0000 7f8619e16440 -1 osd.1 0 log_to_monitors true 2024-09-18T11:24:41.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:41.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:41.117 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:24:41.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:24:41.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:41.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:24:41.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:42.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:42.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:42.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:24:42.566 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:24:42.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:42.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:24:43.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:44.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:44.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:44.064 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:24:44.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:24:44.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:44.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:24:44.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:45.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:45.458 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:24:45.458 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:45.458 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:24:45.458 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:45.458 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:24:46.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:47.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:47.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:47.009 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:24:47.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:24:47.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:47.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:24:47.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:48.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:48.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:48.401 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:24:48.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:24:48.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:48.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:24:48.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:49.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:49.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:49.901 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:24:49.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:24:49.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:49.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:24:50.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:51.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:51.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:51.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:24:51.307 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:24:51.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:51.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:24:51.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:52.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:52.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:52.725 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:24:52.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:24:52.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:52.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:24:53.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:54.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:54.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:54.362 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:24:54.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:24:54.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:54.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:24:54.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:55.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:55.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:55.980 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:24:55.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:24:55.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:55.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:24:56.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:57.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:57.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:57.755 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:24:57.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:24:57.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:57.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:24:58.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:59.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:59.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:59.495 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:24:59.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:24:59.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:59.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:25:00.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:01.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:01.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:01.264 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:25:01.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:25:01.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:01.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:25:02.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:03.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:03.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:03.352 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:25:03.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:25:03.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:03.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:25:03.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:04.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:04.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:04.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:25:04.961 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:25:04.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:04.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:25:05.322 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/50303,v1:127.0.0.1:6811/50303] [v2:127.0.0.1:6812/50303,v1:127.0.0.1:6813/50303] exists,up ba24956b-1eb9-4d1f-96f9-938759a0b7bd 2024-09-18T11:25:05.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:25:05.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:25:05.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:25:05.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2024-09-18T11:25:05.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-18T11:25:05.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:25:05.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:25:05.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:25:05.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-09-18T11:25:05.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-18T11:25:05.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:25:05.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:25:05.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:25:05.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-09-18T11:25:05.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-09-18T11:25:05.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:25:05.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-18T11:25:05.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-18T11:25:05.328 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:25:05.328 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:25:05.328 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:25:05.328 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:25:05.328 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:25:05.328 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:25:05.328 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41135/$cluster-$name.asok' 2024-09-18T11:25:05.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' 2024-09-18T11:25:05.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:25:05.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:25:05.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:25:05.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-18T11:25:05.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-18T11:25:05.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:25:05.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:25:05.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:25:05.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:25:05.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:25:05.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:25:05.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-09-18T11:25:05.329 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:25:05.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c9ceef25-9962-44a7-8a08-7b4da115f0e2 2024-09-18T11:25:05.330 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 c9ceef25-9962-44a7-8a08-7b4da115f0e2 2024-09-18T11:25:05.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c9ceef25-9962-44a7-8a08-7b4da115f0e2' 2024-09-18T11:25:05.330 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:25:05.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRuOpmA1JBFBAAREqGtS1qnDuS3A4Ba9QoQw== 2024-09-18T11:25:05.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRuOpmA1JBFBAAREqGtS1qnDuS3A4Ba9QoQw=="}' 2024-09-18T11:25:05.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c9ceef25-9962-44a7-8a08-7b4da115f0e2 -i td/network-ping/2/new.json 2024-09-18T11:25:05.871 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:25:05.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-09-18T11:25:05.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCRuOpmA1JBFBAAREqGtS1qnDuS3A4Ba9QoQw== --osd-uuid c9ceef25-9962-44a7-8a08-7b4da115f0e2 2024-09-18T11:25:05.909 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:25:05.907+0000 7fa7b1c15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:05.916 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:25:05.915+0000 7fa7b1c15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:05.917 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:25:05.916+0000 7fa7b1c15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:05.917 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:25:05.916+0000 7fa7b1c15440 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-18T11:25:05.918 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:25:05.916+0000 7fa7b1c15440 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-18T11:25:05.918 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:25:05.916+0000 7fa7b1c15440 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-18T11:25:05.956 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:25:05.955+0000 7fa7b1c15440 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-09-18T11:25:12.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-09-18T11:25:12.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:25:12.191 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-18T11:25:12.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:25:12.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:25:12.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:25:12.682 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-18T11:25:12.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=26de567b-cbd5-4f56-97b1-6423ca76f9d2 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:25:12.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:25:12.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:25:12.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:25:12.705 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:25:12.704+0000 7f99f4162440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:12.707 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:25:12.706+0000 7f99f4162440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:12.708 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:25:12.707+0000 7f99f4162440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:13.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:25:13.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:25:13.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:25:13.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:25:13.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:25:13.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:13.054 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:25:13.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:25:13.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:13.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:13.222 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:25:13.221+0000 7f99f4162440 -1 Falling back to public interface 2024-09-18T11:25:13.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:14.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:14.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:14.456 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:25:14.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:25:14.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:14.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:14.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:15.796 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:25:15.794+0000 7f99f4162440 -1 osd.2 0 log_to_monitors true 2024-09-18T11:25:15.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:15.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:15.816 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:25:15.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:25:15.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:15.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:16.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:17.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:17.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:17.297 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:25:17.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:25:17.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:17.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:17.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:18.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:18.714 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:25:18.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:18.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:25:18.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:18.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:19.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:20.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:20.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:20.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:25:20.078 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:25:20.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:20.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:20.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:21.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:21.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:21.620 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:25:21.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:25:21.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:21.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:22.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:23.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:23.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:23.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:25:23.066 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:25:23.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:23.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:23.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:24.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:24.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:24.442 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:25:24.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:25:24.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:24.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:24.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:25.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:25.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:25.833 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:25:25.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:25:25.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:25.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:26.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:27.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:27.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:27.274 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:25:27.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:25:27.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:27.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:27.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:28.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:28.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:28.771 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:25:28.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:25:28.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:28.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:29.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:30.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:30.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:30.224 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:25:30.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:25:30.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:30.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:30.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:31.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:31.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:31.885 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:25:31.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:25:31.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:31.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:32.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:33.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:33.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:33.721 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:25:33.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:25:33.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:33.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:34.480 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:35.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:35.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:35.482 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:25:35.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:25:35.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:35.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:36.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:37.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:37.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:37.393 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:25:37.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:25:37.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:37.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:38.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:39.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:39.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:39.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:25:39.315 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:25:39.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:39.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:25:39.728 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/52060,v1:127.0.0.1:6819/52060] [v2:127.0.0.1:6820/52060,v1:127.0.0.1:6821/52060] exists,up c9ceef25-9962-44a7-8a08-7b4da115f0e2 2024-09-18T11:25:39.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:25:39.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:25:39.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:25:39.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2024-09-18T11:25:44.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2024-09-18T11:25:45.303 INFO:tasks.workunit.client.0.smithi157.stderr:bucket 'dc1' already exists 2024-09-18T11:25:45.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2024-09-18T11:25:45.882 INFO:tasks.workunit.client.0.smithi157.stderr:bucket 'dc2' already exists 2024-09-18T11:25:45.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2024-09-18T11:25:46.501 INFO:tasks.workunit.client.0.smithi157.stderr:bucket 'dc3' already exists 2024-09-18T11:25:46.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2024-09-18T11:25:47.121 INFO:tasks.workunit.client.0.smithi157.stderr:bucket 'rack1' already exists 2024-09-18T11:25:47.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2024-09-18T11:25:47.684 INFO:tasks.workunit.client.0.smithi157.stderr:bucket 'rack2' already exists 2024-09-18T11:25:47.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2024-09-18T11:25:48.167 INFO:tasks.workunit.client.0.smithi157.stderr:bucket 'rack3' already exists 2024-09-18T11:25:48.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2024-09-18T11:25:48.704 INFO:tasks.workunit.client.0.smithi157.stderr:bucket 'host1' already exists 2024-09-18T11:25:48.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2024-09-18T11:25:49.189 INFO:tasks.workunit.client.0.smithi157.stderr:bucket 'host2' already exists 2024-09-18T11:25:49.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2024-09-18T11:25:49.719 INFO:tasks.workunit.client.0.smithi157.stderr:bucket 'host3' already exists 2024-09-18T11:25:49.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2024-09-18T11:25:50.246 INFO:tasks.workunit.client.0.smithi157.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2024-09-18T11:25:50.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2024-09-18T11:25:50.740 INFO:tasks.workunit.client.0.smithi157.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2024-09-18T11:25:50.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2024-09-18T11:25:51.409 INFO:tasks.workunit.client.0.smithi157.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2024-09-18T11:25:51.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2024-09-18T11:25:51.985 INFO:tasks.workunit.client.0.smithi157.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2024-09-18T11:25:51.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2024-09-18T11:25:52.548 INFO:tasks.workunit.client.0.smithi157.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2024-09-18T11:25:52.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2024-09-18T11:25:53.142 INFO:tasks.workunit.client.0.smithi157.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2024-09-18T11:25:53.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2024-09-18T11:25:53.621 INFO:tasks.workunit.client.0.smithi157.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2024-09-18T11:25:53.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2024-09-18T11:25:54.165 INFO:tasks.workunit.client.0.smithi157.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2024-09-18T11:25:54.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2024-09-18T11:25:54.693 INFO:tasks.workunit.client.0.smithi157.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2024-09-18T11:25:54.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2024-09-18T11:25:55.218 INFO:tasks.workunit.client.0.smithi157.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2024-09-18T11:25:55.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2024-09-18T11:25:55.872 INFO:tasks.workunit.client.0.smithi157.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2024-09-18T11:25:55.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2024-09-18T11:25:56.450 INFO:tasks.workunit.client.0.smithi157.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2024-09-18T11:25:56.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2024-09-18T11:25:57.044 INFO:tasks.workunit.client.0.smithi157.stderr:rule myrule already exists 2024-09-18T11:25:57.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2024-09-18T11:25:57.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 16 replicated myrule 2024-09-18T11:25:57.587 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'foo' already exists 2024-09-18T11:25:57.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:25:58.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-09-18T11:25:58.710 INFO:tasks.workunit.client.0.smithi157.stdout:hints = 1 2024-09-18T11:25:58.711 INFO:tasks.workunit.client.0.smithi157.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-09-18T11:25:58.711 INFO:tasks.workunit.client.0.smithi157.stdout:Object prefix: benchmark_data_smithi157_53506 2024-09-18T11:25:58.711 INFO:tasks.workunit.client.0.smithi157.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:25:58.711 INFO:tasks.workunit.client.0.smithi157.stdout: 0 0 0 0 0 0 - 0 2024-09-18T11:25:59.711 INFO:tasks.workunit.client.0.smithi157.stdout: 1 16 53 37 0.144516 0.144531 0.263599 0.341364 2024-09-18T11:26:00.711 INFO:tasks.workunit.client.0.smithi157.stdout: 2 16 117 101 0.197242 0.25 0.244271 0.294945 2024-09-18T11:26:01.711 INFO:tasks.workunit.client.0.smithi157.stdout: 3 16 181 165 0.214819 0.25 0.295611 0.286232 2024-09-18T11:26:02.711 INFO:tasks.workunit.client.0.smithi157.stdout: 4 16 245 229 0.223606 0.25 0.214667 0.274694 2024-09-18T11:26:03.711 INFO:tasks.workunit.client.0.smithi157.stdout: 5 16 309 293 0.228879 0.25 0.235404 0.270154 2024-09-18T11:26:04.711 INFO:tasks.workunit.client.0.smithi157.stdout: 6 16 362 346 0.225233 0.207031 0.277963 0.268926 2024-09-18T11:26:05.711 INFO:tasks.workunit.client.0.smithi157.stdout: 7 16 410 394 0.219839 0.1875 0.501649 0.269692 2024-09-18T11:26:06.711 INFO:tasks.workunit.client.0.smithi157.stdout: 8 16 474 458 0.223605 0.25 0.252333 0.273306 2024-09-18T11:26:07.712 INFO:tasks.workunit.client.0.smithi157.stdout: 9 16 538 522 0.226534 0.25 0.277405 0.271769 2024-09-18T11:26:08.712 INFO:tasks.workunit.client.0.smithi157.stdout: 10 16 598 582 0.227315 0.234375 0.259976 0.270074 2024-09-18T11:26:09.712 INFO:tasks.workunit.client.0.smithi157.stdout:Total time run: 10.2128 2024-09-18T11:26:09.712 INFO:tasks.workunit.client.0.smithi157.stdout:Total writes made: 598 2024-09-18T11:26:09.712 INFO:tasks.workunit.client.0.smithi157.stdout:Write size: 4096 2024-09-18T11:26:09.712 INFO:tasks.workunit.client.0.smithi157.stdout:Object size: 4096 2024-09-18T11:26:09.712 INFO:tasks.workunit.client.0.smithi157.stdout:Bandwidth (MB/sec): 0.228727 2024-09-18T11:26:09.713 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev Bandwidth: 0.0364955 2024-09-18T11:26:09.713 INFO:tasks.workunit.client.0.smithi157.stdout:Max bandwidth (MB/sec): 0.25 2024-09-18T11:26:09.713 INFO:tasks.workunit.client.0.smithi157.stdout:Min bandwidth (MB/sec): 0.144531 2024-09-18T11:26:09.713 INFO:tasks.workunit.client.0.smithi157.stdout:Average IOPS: 58 2024-09-18T11:26:09.713 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev IOPS: 9.34285 2024-09-18T11:26:09.713 INFO:tasks.workunit.client.0.smithi157.stdout:Max IOPS: 64 2024-09-18T11:26:09.713 INFO:tasks.workunit.client.0.smithi157.stdout:Min IOPS: 37 2024-09-18T11:26:09.713 INFO:tasks.workunit.client.0.smithi157.stdout:Average Latency(s): 0.26979 2024-09-18T11:26:09.713 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev Latency(s): 0.0565819 2024-09-18T11:26:09.713 INFO:tasks.workunit.client.0.smithi157.stdout:Max latency(s): 0.528748 2024-09-18T11:26:09.713 INFO:tasks.workunit.client.0.smithi157.stdout:Min latency(s): 0.214436 2024-09-18T11:26:09.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2024-09-18T11:26:19.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2024-09-18T11:26:19.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:26:19.832 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:26:20.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:26:20.189 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-18T11:26:20.190 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-18T11:26:20.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:26:20.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:26:20.190 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:26:20.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836506 2024-09-18T11:26:20.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836506 2024-09-18T11:26:20.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506' 2024-09-18T11:26:20.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:26:20.296 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:26:20.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672982 2024-09-18T11:26:20.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672982 2024-09-18T11:26:20.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506 1-42949672982' 2024-09-18T11:26:20.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:26:20.402 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:26:20.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509452 2024-09-18T11:26:20.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509452 2024-09-18T11:26:20.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506 1-42949672982 2-64424509452' 2024-09-18T11:26:20.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:20.507 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836506 2024-09-18T11:26:20.507 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:20.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:26:20.508 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836506 2024-09-18T11:26:20.508 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:20.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836506 2024-09-18T11:26:20.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-18T11:26:20.510 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836506 2024-09-18T11:26:20.510 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:26:20.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836504 -lt 21474836506 2024-09-18T11:26:20.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:26:21.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:26:21.864 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:26:22.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-18T11:26:22.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:22.223 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672982 2024-09-18T11:26:22.223 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:22.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:26:22.225 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672982 2024-09-18T11:26:22.225 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:22.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672982 2024-09-18T11:26:22.226 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672982 2024-09-18T11:26:22.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-18T11:26:22.226 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:26:22.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-18T11:26:22.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:22.578 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509452 2024-09-18T11:26:22.578 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:22.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:26:22.579 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509452 2024-09-18T11:26:22.579 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:22.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509452 2024-09-18T11:26:22.580 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509452 2024-09-18T11:26:22.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-18T11:26:22.581 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:26:22.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-18T11:26:22.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2024-09-18T11:26:22.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2024-09-18T11:26:23.414 INFO:tasks.workunit.client.0.smithi157.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.336ms); Slow OSD heartbeats on front (longest 0.364ms); 1 pool(s) do not have an application enabled 2024-09-18T11:26:23.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2024-09-18T11:26:23.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2024-09-18T11:26:23.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2024-09-18T11:26:23.891 INFO:tasks.workunit.client.0.smithi157.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.336ms); Slow OSD heartbeats on front (longest 0.364ms); 1 pool(s) do not have an application enabled 2024-09-18T11:26:23.892 INFO:tasks.workunit.client.0.smithi157.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.336ms) 2024-09-18T11:26:23.892 INFO:tasks.workunit.client.0.smithi157.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.336 msec 2024-09-18T11:26:23.892 INFO:tasks.workunit.client.0.smithi157.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.318 msec 2024-09-18T11:26:23.892 INFO:tasks.workunit.client.0.smithi157.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.313 msec 2024-09-18T11:26:23.892 INFO:tasks.workunit.client.0.smithi157.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.308 msec 2024-09-18T11:26:23.892 INFO:tasks.workunit.client.0.smithi157.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.304 msec 2024-09-18T11:26:23.892 INFO:tasks.workunit.client.0.smithi157.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.276 msec possibly improving 2024-09-18T11:26:23.893 INFO:tasks.workunit.client.0.smithi157.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.364ms) 2024-09-18T11:26:23.893 INFO:tasks.workunit.client.0.smithi157.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.364 msec 2024-09-18T11:26:23.893 INFO:tasks.workunit.client.0.smithi157.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.335 msec 2024-09-18T11:26:23.893 INFO:tasks.workunit.client.0.smithi157.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.326 msec 2024-09-18T11:26:23.893 INFO:tasks.workunit.client.0.smithi157.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.319 msec 2024-09-18T11:26:23.893 INFO:tasks.workunit.client.0.smithi157.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.316 msec 2024-09-18T11:26:23.893 INFO:tasks.workunit.client.0.smithi157.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.288 msec 2024-09-18T11:26:23.893 INFO:tasks.workunit.client.0.smithi157.stdout:[WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-18T11:26:23.893 INFO:tasks.workunit.client.0.smithi157.stdout: application not enabled on pool 'foo' 2024-09-18T11:26:23.893 INFO:tasks.workunit.client.0.smithi157.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-18T11:26:23.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2024-09-18T11:26:23.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2024-09-18T11:26:23.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2024-09-18T11:26:23.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2024-09-18T11:26:23.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-09-18T11:26:23.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-18T11:26:23.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:26:23.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-18T11:26:23.907 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:26:23.907 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:26:23.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:26:23.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:26:23.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:26:24.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:26:24.014 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:26:24.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:26:24.015 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:26:24.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:26:24.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:26:24.016 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:26:24.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:26:24.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:26:24.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:26:24.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:26:24.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:26:24.019 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:26:24.019 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-18T11:26:24.038 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:26:24.038 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:24.038 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:26:24.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.41135 2024-09-18T11:26:24.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:26:24.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:26:24.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T11:26:24.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/network-ping 0 2024-09-18T11:26:24.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-18T11:26:24.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T11:26:24.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-18T11:26:24.040 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:26:24.041 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:26:24.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:26:24.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:26:24.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:26:24.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:26:24.042 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:26:24.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:26:24.043 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:26:24.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:26:24.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:26:24.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:26:24.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:26:24.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:26:24.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:26:24.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:26:24.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:26:24.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T11:26:24.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-18T11:26:24.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:26:24.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:24.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41135 2024-09-18T11:26:24.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.41135 2024-09-18T11:26:24.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:26:24.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:26:24.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T11:26:24.052 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T11:26:24.052 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T11:26:24.078 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2024-09-18T11:26:24.078 DEBUG:teuthology.orchestra.run.smithi157:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2024-09-18T11:26:24.140 INFO:tasks.workunit.client.0.smithi157.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T11:26:24.147 INFO:tasks.workunit.client.0.smithi157.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T11:26:24.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T11:26:24.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T11:26:24.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T11:26:24.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T11:26:24.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T11:26:24.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T11:26:24.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T11:26:24.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T11:26:24.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/ok-to-stop 2024-09-18T11:26:24.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2024-09-18T11:26:24.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2024-09-18T11:26:24.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2024-09-18T11:26:24.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2024-09-18T11:26:24.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2024-09-18T11:26:24.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2024-09-18T11:26:24.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2024-09-18T11:26:24.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2024-09-18T11:26:24.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2024-09-18T11:26:24.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2024-09-18T11:26:24.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2024-09-18T11:26:24.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2024-09-18T11:26:24.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2024-09-18T11:26:24.149 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2024-09-18T11:26:24.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none ' 2024-09-18T11:26:24.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none ' 2024-09-18T11:26:24.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none ' 2024-09-18T11:26:24.149 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2024-09-18T11:26:24.149 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T11:26:24.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2024-09-18T11:26:24.151 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_0_osd 2024-09-18T11:26:24.151 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_1_mon_checks 2024-09-18T11:26:24.151 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_2_mons_checks 2024-09-18T11:26:24.151 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_3_mons_checks 2024-09-18T11:26:24.152 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_4_mons_checks 2024-09-18T11:26:24.152 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_5_mons_checks' 2024-09-18T11:26:24.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-18T11:26:24.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-18T11:26:24.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-18T11:26:24.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-18T11:26:24.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-18T11:26:24.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:26:24.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-18T11:26:24.152 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:26:24.152 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:26:24.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:26:24.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:26:24.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:26:24.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:26:24.153 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:26:24.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:26:24.154 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:26:24.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:26:24.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:26:24.154 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:26:24.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:26:24.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:26:24.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:26:24.154 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:26:24.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:26:24.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:26:24.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-18T11:26:24.157 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:26:24.157 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:24.157 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:26:24.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53871 2024-09-18T11:26:24.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:26:24.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:26:24.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-18T11:26:24.159 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:26:24.159 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:24.159 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:26:24.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53871 2024-09-18T11:26:24.162 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:26:24.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T11:26:24.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T11:26:24.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:26:24.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-18T11:26:24.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2024-09-18T11:26:24.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2024-09-18T11:26:24.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-18T11:26:24.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-18T11:26:24.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:26:24.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:26:24.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:26:24.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:26:24.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-18T11:26:24.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-18T11:26:24.523 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:26:24.524 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:26:24.524 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:26:24.524 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:26:24.524 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:24.524 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:26:24.524 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:26:24.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-18T11:26:24.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:26:24.778 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:26:24.778 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:26:24.778 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:26:24.778 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:26:24.778 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:26:24.779 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:26:24.779 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:26:24.779 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:26:24.779 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:26:24.779 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:24.779 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:26:24.779 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.a.asok 2024-09-18T11:26:24.779 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:26:24.779 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.a.asok config get fsid 2024-09-18T11:26:24.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:26:24.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:26:24.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:26:24.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:26:24.855 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:26:24.855 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:26:24.855 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:26:24.855 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:26:24.855 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:26:24.855 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:24.855 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:26:24.855 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.a.asok 2024-09-18T11:26:24.855 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:26:24.855 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.a.asok config get mon_host 2024-09-18T11:26:24.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2024-09-18T11:26:24.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-09-18T11:26:24.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:26:24.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:26:24.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:26:24.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-09-18T11:26:24.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:26:25.763 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:26:25.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:26:25.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:26:25.764 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:26:25.764 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:25.764 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:26:25.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:26:25.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:26:25.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:26:25.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2024-09-18T11:26:25.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-18T11:26:25.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:26:25.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:26:25.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:26:25.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-09-18T11:26:25.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-18T11:26:25.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:26:25.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:26:25.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:26:25.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-09-18T11:26:25.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-09-18T11:26:25.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:26:25.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:26:25.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-18T11:26:25.790 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:26:25.790 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:26:25.790 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:26:25.790 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:26:25.790 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:25.790 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:26:25.790 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:26:25.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:26:25.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:26:25.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:26:25.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:26:25.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-18T11:26:25.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-18T11:26:25.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:26:25.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:26:25.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:26:25.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:26:25.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:26:25.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:26:25.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-09-18T11:26:25.791 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:26:25.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=32bea45c-bca8-46c5-811a-cbefddfb5eaf 2024-09-18T11:26:25.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 32bea45c-bca8-46c5-811a-cbefddfb5eaf' 2024-09-18T11:26:25.792 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 32bea45c-bca8-46c5-811a-cbefddfb5eaf 2024-09-18T11:26:25.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:26:25.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhuOpm2BHDLxAA2LAbBllm2lEB5dc4ENs5ZQ== 2024-09-18T11:26:25.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhuOpm2BHDLxAA2LAbBllm2lEB5dc4ENs5ZQ=="}' 2024-09-18T11:26:25.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 32bea45c-bca8-46c5-811a-cbefddfb5eaf -i td/ok-to-stop/0/new.json 2024-09-18T11:26:26.116 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:26:26.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-09-18T11:26:26.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhuOpm2BHDLxAA2LAbBllm2lEB5dc4ENs5ZQ== --osd-uuid 32bea45c-bca8-46c5-811a-cbefddfb5eaf 2024-09-18T11:26:26.149 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:26:26.147+0000 7fac45d1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:26.161 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:26:26.160+0000 7fac45d1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:26.162 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:26:26.161+0000 7fac45d1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:26.163 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:26:26.162+0000 7fac45d1d440 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-18T11:26:26.163 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:26:26.162+0000 7fac45d1d440 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-18T11:26:26.163 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:26:26.162+0000 7fac45d1d440 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-18T11:26:26.202 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:26:26.200+0000 7fac45d1d440 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-09-18T11:26:31.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-09-18T11:26:31.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:26:31.522 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:26:31.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:26:31.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:26:31.998 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:26:31.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:26:31.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:26:31.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:26:31.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:26:31.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:26:32.019 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:26:32.017+0000 7fafb5323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:32.021 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:26:32.019+0000 7fafb5323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:32.022 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:26:32.021+0000 7fafb5323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:32.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:26:32.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:26:32.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:26:32.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:26:32.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:26:32.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:32.384 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:26:32.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:26:32.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:32.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:32.537 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:26:32.536+0000 7fafb5323440 -1 Falling back to public interface 2024-09-18T11:26:32.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:33.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:33.752 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:26:33.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:33.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:26:33.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:33.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:34.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:35.056 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:26:35.055+0000 7fafb5323440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:26:35.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:35.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:35.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:26:35.104 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:26:35.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:35.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:35.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:36.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:36.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:36.618 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:26:36.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:26:36.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:36.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:37.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:38.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:38.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:38.037 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:26:38.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:26:38.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:38.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:38.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:39.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:39.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:39.402 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:26:39.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:26:39.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:39.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:39.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:40.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:40.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:40.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:26:40.850 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:26:40.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:40.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:41.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:42.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:42.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:42.207 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:26:42.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:26:42.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:42.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:42.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:43.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:43.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:43.721 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:26:43.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:26:43.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:43.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:44.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:45.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:45.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:45.084 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:26:45.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:26:45.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:45.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:45.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:46.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:46.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:46.455 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:26:46.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:26:46.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:46.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:47.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:48.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:48.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:48.259 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:26:48.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:26:48.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:48.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:48.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:49.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:49.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:49.759 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:26:49.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:26:49.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:49.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:50.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:51.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:51.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:51.565 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:26:51.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:26:51.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:51.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:52.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:53.339 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:53.340 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:53.340 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:26:53.340 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:26:53.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:53.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:53.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:54.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:54.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:54.758 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:26:54.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:26:54.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:54.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:55.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:56.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:56.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:56.512 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:26:56.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:26:56.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:56.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:56.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:57.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:57.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:57.868 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:26:57.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:26:57.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:57.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:26:58.256 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/54962,v1:127.0.0.1:6803/54962] [v2:127.0.0.1:6804/54962,v1:127.0.0.1:6805/54962] exists,up 32bea45c-bca8-46c5-811a-cbefddfb5eaf 2024-09-18T11:26:58.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:26:58.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:26:58.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:26:58.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2024-09-18T11:26:58.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-09-18T11:26:58.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-18T11:26:58.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=a 2024-09-18T11:26:58.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-18T11:26:58.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/a 2024-09-18T11:26:58.259 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-18T11:26:58.259 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:26:58.259 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:26:58.259 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:26:58.259 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:58.259 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:26:58.259 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:26:58.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-09-18T11:26:58.278 INFO:tasks.workunit.client.0.smithi157.stdout:starting mds.a at 2024-09-18T11:26:58.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2024-09-18T11:26:58.870 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'meta' already exists 2024-09-18T11:26:58.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2024-09-18T11:26:59.511 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'data' already exists 2024-09-18T11:26:59.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2024-09-18T11:27:00.418 INFO:tasks.workunit.client.0.smithi157.stderr:filesystem 'myfs' already exists 2024-09-18T11:27:00.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2024-09-18T11:27:05.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2024-09-18T11:27:05.742 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-09-18T11:27:05.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2024-09-18T11:27:06.052 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-09-18T11:27:06.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2024-09-18T11:27:06.525 INFO:tasks.workunit.client.0.smithi157.stderr:should be safe to stop [dne] 2024-09-18T11:27:06.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2024-09-18T11:27:06.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-09-18T11:27:06.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-18T11:27:06.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=b 2024-09-18T11:27:06.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-18T11:27:06.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/b 2024-09-18T11:27:06.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-18T11:27:06.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:27:06.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:27:06.537 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:27:06.538 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:27:06.538 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:27:06.538 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:27:06.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-09-18T11:27:06.556 INFO:tasks.workunit.client.0.smithi157.stdout:starting mds.b at 2024-09-18T11:27:06.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2024-09-18T11:27:11.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2024-09-18T11:27:12.030 INFO:tasks.workunit.client.0.smithi157.stderr:should be safe to stop [a] 2024-09-18T11:27:12.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2024-09-18T11:27:12.501 INFO:tasks.workunit.client.0.smithi157.stderr:should be safe to stop [b] 2024-09-18T11:27:12.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2024-09-18T11:27:12.811 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-09-18T11:27:12.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2024-09-18T11:27:13.281 INFO:tasks.workunit.client.0.smithi157.stderr:should be safe to stop [a,dne1,dne2] 2024-09-18T11:27:13.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2024-09-18T11:27:13.755 INFO:tasks.workunit.client.0.smithi157.stderr:should be safe to stop [b,dne] 2024-09-18T11:27:13.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2024-09-18T11:27:14.070 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-09-18T11:27:14.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2024-09-18T11:27:14.548 INFO:tasks.workunit.client.0.smithi157.stderr:should be safe to stop [dne1,dne2] 2024-09-18T11:27:14.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2024-09-18T11:27:14.559 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:27:14.559 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:27:14.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:27:14.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:27:14.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:27:14.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:27:14.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-18T11:27:14.665 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:27:14.665 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:27:14.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:27:14.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:27:14.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:27:14.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:27:14.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-18T11:27:14.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-18T11:27:14.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:27:14.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-18T11:27:14.773 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:27:14.773 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:27:14.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:27:14.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:27:14.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:27:14.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:27:14.778 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:27:14.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:27:14.778 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:27:14.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:27:14.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:27:14.780 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:27:14.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:27:14.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:27:14.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:27:14.781 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:27:14.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:27:14.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:27:14.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-18T11:27:14.789 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:27:14.789 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:27:14.789 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:27:14.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53871 2024-09-18T11:27:14.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:27:14.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:27:14.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-18T11:27:14.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-18T11:27:14.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-18T11:27:14.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-18T11:27:14.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-18T11:27:14.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:27:14.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-18T11:27:14.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:27:14.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:27:14.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:27:14.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:27:14.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:27:14.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:27:14.793 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:27:14.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:27:14.793 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:27:14.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:27:14.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:27:14.794 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:27:14.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:27:14.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:27:14.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:27:14.796 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:27:14.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:27:14.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:27:14.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-18T11:27:14.799 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:27:14.799 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:27:14.799 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:27:14.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53871 2024-09-18T11:27:14.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:27:14.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:27:14.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-18T11:27:14.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:27:14.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:27:14.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:27:14.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53871 2024-09-18T11:27:14.803 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:27:14.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:27:14.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:27:14.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-18T11:27:14.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2024-09-18T11:27:14.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2024-09-18T11:27:14.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-18T11:27:14.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-18T11:27:14.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:27:14.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:27:14.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:27:14.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:27:14.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-18T11:27:14.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-18T11:27:15.181 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:27:15.181 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:27:15.181 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:27:15.182 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:27:15.182 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:27:15.182 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:27:15.182 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:27:15.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-18T11:27:15.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:27:15.433 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:27:15.433 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:27:15.433 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:27:15.433 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:27:15.434 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:27:15.434 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:27:15.434 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:27:15.434 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:27:15.434 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:27:15.434 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:27:15.434 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:27:15.434 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.a.asok 2024-09-18T11:27:15.435 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:27:15.435 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.a.asok config get fsid 2024-09-18T11:27:15.510 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:27:15.510 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:27:15.510 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:27:15.510 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:27:15.510 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:27:15.511 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:27:15.511 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:27:15.511 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:27:15.511 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:27:15.511 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:27:15.511 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:27:15.511 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.a.asok 2024-09-18T11:27:15.511 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:27:15.511 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.a.asok config get mon_host 2024-09-18T11:27:15.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2024-09-18T11:27:15.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-09-18T11:27:15.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:27:15.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:27:15.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:27:15.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-09-18T11:27:15.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:27:16.428 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:27:16.428 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:27:16.428 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:27:16.429 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:27:16.429 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:27:16.429 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:27:16.429 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:27:16.429 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:27:16.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:27:16.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2024-09-18T11:27:16.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-18T11:27:16.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:27:16.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:27:16.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:27:16.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-09-18T11:27:16.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-18T11:27:16.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:27:16.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:27:16.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:27:16.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-09-18T11:27:16.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-09-18T11:27:16.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:27:16.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:27:16.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-18T11:27:16.455 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:27:16.456 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:27:16.456 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:27:16.456 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:27:16.456 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:27:16.456 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:27:16.456 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:27:16.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:27:16.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:27:16.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:27:16.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:27:16.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-18T11:27:16.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-18T11:27:16.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:27:16.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:27:16.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:27:16.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:27:16.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:27:16.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:27:16.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-09-18T11:27:16.457 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:27:16.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4b9eff23-457a-499d-878b-f7889573a21c 2024-09-18T11:27:16.458 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4b9eff23-457a-499d-878b-f7889573a21c' 2024-09-18T11:27:16.458 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 4b9eff23-457a-499d-878b-f7889573a21c 2024-09-18T11:27:16.458 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:27:16.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUuepmbk7aGxAAeJYLsdLXCLHrFyVL6u9Ucg== 2024-09-18T11:27:16.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUuepmbk7aGxAAeJYLsdLXCLHrFyVL6u9Ucg=="}' 2024-09-18T11:27:16.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4b9eff23-457a-499d-878b-f7889573a21c -i td/ok-to-stop/0/new.json 2024-09-18T11:27:16.771 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:27:16.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-09-18T11:27:16.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUuepmbk7aGxAAeJYLsdLXCLHrFyVL6u9Ucg== --osd-uuid 4b9eff23-457a-499d-878b-f7889573a21c 2024-09-18T11:27:16.804 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:16.803+0000 7f9747645440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:16.817 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:16.815+0000 7f9747645440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:16.818 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:16.817+0000 7f9747645440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:16.818 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:16.817+0000 7f9747645440 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-18T11:27:16.818 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:16.817+0000 7f9747645440 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-18T11:27:16.819 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:16.817+0000 7f9747645440 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-18T11:27:16.857 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:16.855+0000 7f9747645440 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-09-18T11:27:22.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-09-18T11:27:22.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:27:22.212 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:27:22.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:27:22.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:27:22.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:27:22.690 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:27:22.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:27:22.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:27:22.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:27:22.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:27:22.711 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:22.709+0000 7f600723d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:22.712 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:22.711+0000 7f600723d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:22.713 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:22.712+0000 7f600723d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:23.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:27:23.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:27:23.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:27:23.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:27:23.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:27:23.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:23.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:27:23.041 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:27:23.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:23.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:23.242 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:23.241+0000 7f600723d440 -1 Falling back to public interface 2024-09-18T11:27:23.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:24.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:24.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:24.411 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:27:24.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:27:24.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:24.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:24.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:25.733 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:25.731+0000 7f600723d440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:27:25.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:25.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:25.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:27:25.761 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:27:25.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:25.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:26.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:27.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:27.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:27.116 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:27:27.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:27:27.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:27.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:27.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:28.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:28.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:28.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:27:28.588 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:27:28.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:28.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:28.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:29.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:29.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:29.997 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:27:29.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:27:29.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:29.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:30.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:31.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:31.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:31.476 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:27:31.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:27:31.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:31.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:31.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:32.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:32.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:32.838 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:27:32.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:27:32.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:32.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:33.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:34.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:34.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:34.251 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:27:34.252 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:27:34.252 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:34.252 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:34.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:35.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:35.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:35.774 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:27:35.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:27:35.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:35.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:36.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:37.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:37.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:37.108 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:27:37.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:27:37.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:37.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:38.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:39.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:39.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:39.126 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:27:39.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:27:39.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:39.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:39.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:40.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:40.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:40.659 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:27:40.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:27:40.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:40.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:41.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:42.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:42.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:42.454 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:27:42.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:27:42.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:42.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:43.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:44.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:44.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:44.152 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:27:44.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:27:44.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:44.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:44.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:45.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:45.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:45.580 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:27:45.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:27:45.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:45.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:46.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:47.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:47.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:47.230 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:27:47.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:27:47.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:47.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:47.706 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:48.707 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:48.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:48.708 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:27:48.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:27:48.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:48.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:49.059 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/57384,v1:127.0.0.1:6803/57384] [v2:127.0.0.1:6804/57384,v1:127.0.0.1:6805/57384] exists,up 4b9eff23-457a-499d-878b-f7889573a21c 2024-09-18T11:27:49.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:27:49.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:27:49.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:27:49.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2024-09-18T11:27:49.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-18T11:27:49.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:27:49.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:27:49.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:27:49.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/1 2024-09-18T11:27:49.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-18T11:27:49.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:27:49.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:27:49.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:27:49.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2024-09-18T11:27:49.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2024-09-18T11:27:49.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:27:49.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:27:49.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-18T11:27:49.064 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:27:49.064 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:27:49.064 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:27:49.064 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:27:49.064 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:27:49.064 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:27:49.064 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:27:49.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:27:49.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:27:49.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:27:49.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:27:49.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-18T11:27:49.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-18T11:27:49.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:27:49.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:27:49.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:27:49.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:27:49.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:27:49.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:27:49.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/1 2024-09-18T11:27:49.065 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:27:49.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c8964fa9-da97-4729-8d13-607106a1a20e 2024-09-18T11:27:49.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c8964fa9-da97-4729-8d13-607106a1a20e' 2024-09-18T11:27:49.066 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 c8964fa9-da97-4729-8d13-607106a1a20e 2024-09-18T11:27:49.066 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:27:49.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA1uepmux98BBAAgswDq7weXRU3sLI0Nx7fGQ== 2024-09-18T11:27:49.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA1uepmux98BBAAgswDq7weXRU3sLI0Nx7fGQ=="}' 2024-09-18T11:27:49.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c8964fa9-da97-4729-8d13-607106a1a20e -i td/ok-to-stop/1/new.json 2024-09-18T11:27:49.569 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:27:49.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/1/new.json 2024-09-18T11:27:49.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1uepmux98BBAAgswDq7weXRU3sLI0Nx7fGQ== --osd-uuid c8964fa9-da97-4729-8d13-607106a1a20e 2024-09-18T11:27:49.603 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:49.601+0000 7f5158e73440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:49.604 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:49.603+0000 7f5158e73440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:49.605 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:49.604+0000 7f5158e73440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:49.606 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:49.605+0000 7f5158e73440 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-09-18T11:27:49.606 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:49.605+0000 7f5158e73440 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-09-18T11:27:49.606 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:49.605+0000 7f5158e73440 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-09-18T11:27:49.643 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:49.642+0000 7f5158e73440 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2024-09-18T11:27:55.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/1/keyring 2024-09-18T11:27:55.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:27:55.785 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T11:27:55.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:27:55.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:27:56.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:27:56.272 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T11:27:56.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:27:56.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:27:56.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:27:56.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:27:56.292 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:56.290+0000 7f3518616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:56.316 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:56.314+0000 7f3518616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:56.316 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:56.315+0000 7f3518616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:56.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:27:56.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:27:56.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:27:56.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:27:56.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:27:56.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:56.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:27:56.663 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:27:56.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:56.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:27:56.851 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:56.850+0000 7f3518616440 -1 Falling back to public interface 2024-09-18T11:27:57.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:58.028 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:58.028 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:27:58.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:58.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:27:58.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:58.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:27:58.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:59.328 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:27:59.326+0000 7f3518616440 -1 osd.1 0 log_to_monitors true 2024-09-18T11:27:59.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:59.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:59.384 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:27:59.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:27:59.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:59.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:00.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:01.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:01.040 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:28:01.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:01.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:28:01.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:01.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:01.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:02.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:02.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:02.435 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:28:02.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:28:02.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:02.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:02.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:03.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:03.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:03.966 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:28:03.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:28:03.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:03.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:04.480 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:05.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:05.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:05.482 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:28:05.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:28:05.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:05.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:05.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:06.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:06.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:06.853 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:28:06.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:28:06.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:06.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:07.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:08.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:08.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:08.357 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:28:08.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:28:08.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:08.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:08.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:09.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:09.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:09.780 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:28:09.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:28:09.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:09.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:10.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:11.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:11.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:11.303 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:28:11.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:28:11.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:11.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:11.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:12.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:12.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:12.868 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:28:12.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:28:12.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:12.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:14.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:15.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:15.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:15.046 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:28:15.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:28:15.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:15.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:15.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:16.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:16.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:16.992 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:28:16.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:28:16.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:16.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:17.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:18.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:18.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:18.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:28:18.926 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:28:18.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:18.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:19.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:20.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:20.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:20.728 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:28:20.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:28:20.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:20.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:21.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:22.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:22.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:22.778 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:28:22.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:28:22.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:22.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:23.139 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/59147,v1:127.0.0.1:6811/59147] [v2:127.0.0.1:6812/59147,v1:127.0.0.1:6813/59147] exists,up c8964fa9-da97-4729-8d13-607106a1a20e 2024-09-18T11:28:23.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:28:23.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:28:23.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:28:23.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2024-09-18T11:28:23.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-18T11:28:23.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:28:23.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:28:23.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:28:23.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/2 2024-09-18T11:28:23.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-18T11:28:23.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:28:23.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:28:23.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:28:23.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2024-09-18T11:28:23.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2024-09-18T11:28:23.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:28:23.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:28:23.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-18T11:28:23.145 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:28:23.145 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:28:23.145 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:28:23.145 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:28:23.145 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:28:23.145 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:28:23.145 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:28:23.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:28:23.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:28:23.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:28:23.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:28:23.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-18T11:28:23.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-18T11:28:23.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:28:23.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:28:23.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:28:23.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:28:23.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:28:23.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:28:23.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/2 2024-09-18T11:28:23.146 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:28:23.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bb612c75-fbb8-427c-b13f-52321b4ef47e 2024-09-18T11:28:23.146 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 bb612c75-fbb8-427c-b13f-52321b4ef47e 2024-09-18T11:28:23.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bb612c75-fbb8-427c-b13f-52321b4ef47e' 2024-09-18T11:28:23.147 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:28:23.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBXuepm+ptFCRAAqE4ADyhCjFpqEoeH6zmxGA== 2024-09-18T11:28:23.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBXuepm+ptFCRAAqE4ADyhCjFpqEoeH6zmxGA=="}' 2024-09-18T11:28:23.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bb612c75-fbb8-427c-b13f-52321b4ef47e -i td/ok-to-stop/2/new.json 2024-09-18T11:28:23.610 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:28:23.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/2/new.json 2024-09-18T11:28:23.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXuepm+ptFCRAAqE4ADyhCjFpqEoeH6zmxGA== --osd-uuid bb612c75-fbb8-427c-b13f-52321b4ef47e 2024-09-18T11:28:23.644 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:23.643+0000 7f2a7c647440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:23.646 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:23.644+0000 7f2a7c647440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:23.647 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:23.645+0000 7f2a7c647440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:23.647 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:23.646+0000 7f2a7c647440 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-09-18T11:28:23.647 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:23.646+0000 7f2a7c647440 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-09-18T11:28:23.647 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:23.646+0000 7f2a7c647440 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-09-18T11:28:23.692 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:23.691+0000 7f2a7c647440 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2024-09-18T11:28:29.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/2/keyring 2024-09-18T11:28:29.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:28:29.573 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-18T11:28:29.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:28:29.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:28:30.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:28:30.068 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-18T11:28:30.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:28:30.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:28:30.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:28:30.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:28:30.088 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:30.087+0000 7f411c41a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:30.116 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:30.114+0000 7f411c41a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:30.117 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:30.115+0000 7f411c41a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:30.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:28:30.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:28:30.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:28:30.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:28:30.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:28:30.457 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:28:30.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:30.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:28:30.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:30.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:30.656 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:30.655+0000 7f411c41a440 -1 Falling back to public interface 2024-09-18T11:28:30.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:31.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:31.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:31.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:28:31.807 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:28:31.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:31.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:32.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:33.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:33.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:33.160 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:28:33.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:28:33.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:33.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:33.202 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:33.201+0000 7f411c41a440 -1 osd.2 0 log_to_monitors true 2024-09-18T11:28:33.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:34.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:34.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:34.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:28:34.516 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:28:34.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:34.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:34.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:35.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:35.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:35.934 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:28:35.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:28:35.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:35.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:36.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:37.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:37.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:37.357 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:28:37.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:28:37.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:37.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:37.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:38.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:38.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:38.764 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:28:38.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:28:38.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:38.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:39.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:40.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:40.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:40.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:28:40.121 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:28:40.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:40.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:40.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:41.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:41.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:41.494 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:28:41.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:28:41.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:41.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:41.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:42.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:42.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:42.969 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:28:42.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:28:42.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:42.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:43.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:44.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:44.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:44.399 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:28:44.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:28:44.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:44.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:44.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:45.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:45.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:45.785 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:28:45.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:28:45.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:45.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:46.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:47.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:47.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:47.576 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:28:47.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:28:47.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:47.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:48.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:49.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:49.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:49.461 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:28:49.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:28:49.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:49.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:50.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:51.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:51.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:51.380 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:28:51.381 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:28:51.381 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:51.381 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:52.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:53.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:53.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:53.052 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:28:53.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:28:53.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:53.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:53.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:54.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:54.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:54.935 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:28:54.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:28:54.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:54.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:55.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:56.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:56.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:56.558 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:28:56.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:28:56.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:56.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:56.952 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/60876,v1:127.0.0.1:6819/60876] [v2:127.0.0.1:6820/60876,v1:127.0.0.1:6821/60876] exists,up bb612c75-fbb8-427c-b13f-52321b4ef47e 2024-09-18T11:28:56.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:28:56.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:28:56.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:28:56.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2024-09-18T11:28:56.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-18T11:28:56.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:28:56.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T11:28:56.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:28:56.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/3 2024-09-18T11:28:56.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-18T11:28:56.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:28:56.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:28:56.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:28:56.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2024-09-18T11:28:56.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2024-09-18T11:28:56.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:28:56.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:28:56.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-18T11:28:56.957 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:28:56.957 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:28:56.957 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:28:56.958 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:28:56.958 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:28:56.958 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:28:56.958 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:28:56.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:28:56.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:28:56.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:28:56.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:28:56.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-18T11:28:56.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-18T11:28:56.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:28:56.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:28:56.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:28:56.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:28:56.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:28:56.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:28:56.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/3 2024-09-18T11:28:56.959 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:28:56.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6b8f5965-1113-488c-b698-1f714a01200e 2024-09-18T11:28:56.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 6b8f5965-1113-488c-b698-1f714a01200e' 2024-09-18T11:28:56.959 INFO:tasks.workunit.client.0.smithi157.stdout:add osd3 6b8f5965-1113-488c-b698-1f714a01200e 2024-09-18T11:28:56.960 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:28:56.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB4uepmPqqjORAAc5fx7i4GdrUFM58oIkc36w== 2024-09-18T11:28:56.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB4uepmPqqjORAAc5fx7i4GdrUFM58oIkc36w=="}' 2024-09-18T11:28:56.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6b8f5965-1113-488c-b698-1f714a01200e -i td/ok-to-stop/3/new.json 2024-09-18T11:28:57.511 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:28:57.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/3/new.json 2024-09-18T11:28:57.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB4uepmPqqjORAAc5fx7i4GdrUFM58oIkc36w== --osd-uuid 6b8f5965-1113-488c-b698-1f714a01200e 2024-09-18T11:28:57.546 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:57.545+0000 7f773260d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:57.561 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:57.560+0000 7f773260d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:57.562 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:57.561+0000 7f773260d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:57.563 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:57.561+0000 7f773260d440 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-09-18T11:28:57.563 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:57.561+0000 7f773260d440 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-09-18T11:28:57.563 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:57.561+0000 7f773260d440 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-09-18T11:28:57.637 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:28:57.635+0000 7f773260d440 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2024-09-18T11:29:03.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/3/keyring 2024-09-18T11:29:03.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:29:03.784 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd3 key to auth repository 2024-09-18T11:29:03.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T11:29:03.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:29:04.281 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.3 2024-09-18T11:29:04.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T11:29:04.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:29:04.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:29:04.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:29:04.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:29:04.301 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:29:04.300+0000 7f4234a21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:29:04.303 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:29:04.302+0000 7f4234a21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:29:04.304 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:29:04.303+0000 7f4234a21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:29:04.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T11:29:04.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:29:04.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T11:29:04.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:29:04.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:29:04.633 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:04.633 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:29:04.633 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:29:04.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:04.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:04.821 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:29:04.820+0000 7f4234a21440 -1 Falling back to public interface 2024-09-18T11:29:04.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:06.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:06.000 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:29:06.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:06.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:29:06.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:06.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:06.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:07.325 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:29:07.322+0000 7f4234a21440 -1 osd.3 0 log_to_monitors true 2024-09-18T11:29:07.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:07.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:07.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:29:07.358 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:29:07.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:07.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:07.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:08.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:08.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:08.793 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:29:08.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:29:08.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:08.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:09.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:10.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:10.299 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:29:10.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:10.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:29:10.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:10.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:10.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:11.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:11.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:11.776 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:29:11.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:29:11.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:11.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:12.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:13.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:13.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:13.311 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:29:13.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:29:13.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:13.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:13.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:14.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:14.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:14.676 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:29:14.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:29:14.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:14.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:15.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:16.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:16.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:16.220 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:29:16.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:29:16.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:16.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:16.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:17.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:17.580 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:29:17.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:17.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:29:17.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:17.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:17.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:18.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:18.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:18.941 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:29:18.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:29:18.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:18.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:19.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:20.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:20.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:20.317 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:29:20.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:29:20.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:20.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:21.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:22.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:22.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:22.452 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:29:22.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:29:22.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:22.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:23.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:24.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:24.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:24.051 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:29:24.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:29:24.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:24.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:25.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:26.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:26.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:26.405 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:29:26.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:29:26.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:26.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:27.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:28.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:28.331 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:29:28.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:28.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:29:28.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:28.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:29.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:30.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:30.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:30.040 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:29:30.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:29:30.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:30.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:30.881 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:31.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:31.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:31.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:29:31.883 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:29:31.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:31.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:32.336 INFO:tasks.workunit.client.0.smithi157.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/62630,v1:127.0.0.1:6827/62630] [v2:127.0.0.1:6828/62630,v1:127.0.0.1:6829/62630] exists,up 6b8f5965-1113-488c-b698-1f714a01200e 2024-09-18T11:29:32.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:29:32.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:29:32.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:29:32.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-09-18T11:29:32.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2024-09-18T11:29:34.033 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'ec' already exists 2024-09-18T11:29:34.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2024-09-18T11:29:34.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:29:34.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:29:34.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:29:34.045 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:29:34.046 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:29:34.046 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:29:34.046 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:29:34.046 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:29:34.046 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:29:34.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:29:34.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:29:34.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:29:34.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:29:34.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:29:34.131 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:29:34.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:29:34.723 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-18T11:29:34.723 INFO:tasks.workunit.client.0.smithi157.stderr:2 2024-09-18T11:29:34.723 INFO:tasks.workunit.client.0.smithi157.stderr:3' 2024-09-18T11:29:34.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:29:34.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:34.723 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:29:34.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836505 2024-09-18T11:29:34.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836505 2024-09-18T11:29:34.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505' 2024-09-18T11:29:34.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:34.923 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:29:35.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672978 2024-09-18T11:29:35.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672978 2024-09-18T11:29:35.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672978' 2024-09-18T11:29:35.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:35.230 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:29:35.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542158 2024-09-18T11:29:35.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542158 2024-09-18T11:29:35.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672978 2-60129542158' 2024-09-18T11:29:35.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:35.451 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:29:35.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378632 2024-09-18T11:29:35.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378632 2024-09-18T11:29:35.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672978 2-60129542158 3-81604378632' 2024-09-18T11:29:35.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:35.708 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836505 2024-09-18T11:29:35.708 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:35.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:29:35.709 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836505 2024-09-18T11:29:35.709 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:35.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836505 2024-09-18T11:29:35.711 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836505 2024-09-18T11:29:35.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-18T11:29:35.711 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:29:36.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-18T11:29:36.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:36.254 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672978 2024-09-18T11:29:36.254 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:36.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:29:36.255 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672978 2024-09-18T11:29:36.256 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:36.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672978 2024-09-18T11:29:36.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-18T11:29:36.257 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672978 2024-09-18T11:29:36.257 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:29:36.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-18T11:29:36.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:36.608 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-60129542158 2024-09-18T11:29:36.608 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:36.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:29:36.609 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-60129542158 2024-09-18T11:29:36.610 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:36.610 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 60129542158 2024-09-18T11:29:36.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542158 2024-09-18T11:29:36.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 60129542158' 2024-09-18T11:29:36.611 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:29:36.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542158 -lt 60129542158 2024-09-18T11:29:36.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:36.996 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-81604378632 2024-09-18T11:29:36.996 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:36.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:29:36.997 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-81604378632 2024-09-18T11:29:36.997 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:36.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378632 2024-09-18T11:29:36.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 81604378632' 2024-09-18T11:29:36.999 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.3 seq 81604378632 2024-09-18T11:29:36.999 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:29:37.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378632 -lt 81604378632 2024-09-18T11:29:37.345 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:29:37.345 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:29:37.345 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:29:37.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-18T11:29:37.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:29:37.812 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:29:37.812 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:29:37.812 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:29:37.812 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:29:37.812 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:29:37.812 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:29:38.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-18T11:29:38.164 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:29:38.164 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:29:38.164 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:29:38.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-18T11:29:38.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:29:38.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:29:38.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2024-09-18T11:29:39.258 INFO:tasks.workunit.client.0.smithi157.stderr:set pool 1 min_size to 3 2024-09-18T11:29:39.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2024-09-18T11:29:39.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:29:39.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:29:39.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:29:39.269 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:29:39.270 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:29:39.270 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:29:39.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:29:39.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:29:39.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:29:39.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:29:39.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:29:39.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:29:39.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:29:39.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:29:39.355 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:29:39.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:29:39.714 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-18T11:29:39.714 INFO:tasks.workunit.client.0.smithi157.stderr:2 2024-09-18T11:29:39.714 INFO:tasks.workunit.client.0.smithi157.stderr:3' 2024-09-18T11:29:39.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:29:39.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:39.714 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:29:39.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836508 2024-09-18T11:29:39.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836508 2024-09-18T11:29:39.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508' 2024-09-18T11:29:39.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:39.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:29:39.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672981 2024-09-18T11:29:39.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672981 2024-09-18T11:29:39.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-42949672981' 2024-09-18T11:29:39.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:39.958 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:29:40.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542161 2024-09-18T11:29:40.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542161 2024-09-18T11:29:40.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-42949672981 2-60129542161' 2024-09-18T11:29:40.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:40.069 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:29:40.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378635 2024-09-18T11:29:40.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378635 2024-09-18T11:29:40.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-42949672981 2-60129542161 3-81604378635' 2024-09-18T11:29:40.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:40.173 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836508 2024-09-18T11:29:40.173 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:40.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:29:40.174 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836508 2024-09-18T11:29:40.174 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:40.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836508 2024-09-18T11:29:40.175 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836508 2024-09-18T11:29:40.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-18T11:29:40.175 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:29:40.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-18T11:29:40.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:40.517 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672981 2024-09-18T11:29:40.517 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:40.518 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:29:40.518 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672981 2024-09-18T11:29:40.518 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:40.519 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672981 2024-09-18T11:29:40.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672981 2024-09-18T11:29:40.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-18T11:29:40.520 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:29:40.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-18T11:29:40.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:40.871 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-60129542161 2024-09-18T11:29:40.871 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:40.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:29:40.872 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-60129542161 2024-09-18T11:29:40.872 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:40.873 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 60129542161 2024-09-18T11:29:40.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542161 2024-09-18T11:29:40.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 60129542161' 2024-09-18T11:29:40.873 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:29:41.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542159 -lt 60129542161 2024-09-18T11:29:41.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:29:42.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:29:42.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:29:42.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542162 -lt 60129542161 2024-09-18T11:29:42.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:42.582 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-81604378635 2024-09-18T11:29:42.582 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:42.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:29:42.583 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-81604378635 2024-09-18T11:29:42.583 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:42.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378635 2024-09-18T11:29:42.584 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.3 seq 81604378635 2024-09-18T11:29:42.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 81604378635' 2024-09-18T11:29:42.585 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:29:42.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378636 -lt 81604378635 2024-09-18T11:29:42.963 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:29:42.963 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:29:42.964 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:29:43.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-18T11:29:43.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:29:43.436 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:29:43.436 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:29:43.436 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:29:43.436 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:29:43.436 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:29:43.436 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:29:43.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-18T11:29:43.825 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:29:43.826 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:29:43.826 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:29:44.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-18T11:29:44.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:29:44.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:29:44.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2024-09-18T11:29:44.575 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:29:44.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2024-09-18T11:29:44.922 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:29:44.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2024-09-18T11:29:45.265 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:29:45.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2024-09-18T11:29:45.607 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:29:45.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-18T11:29:45.856 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:29:45.857 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-18T11:29:45.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-09-18T11:29:46.099 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:29:46.099 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-18T11:29:46.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-09-18T11:29:46.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2024-09-18T11:29:46.447 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:29:46.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2024-09-18T11:29:46.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2024-09-18T11:29:46.795 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:29:46.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2024-09-18T11:29:47.632 INFO:tasks.workunit.client.0.smithi157.stderr:set pool 1 min_size to 2 2024-09-18T11:29:47.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2024-09-18T11:29:47.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:29:47.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:29:47.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:29:47.643 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:29:47.644 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:29:47.644 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:29:47.644 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:29:47.644 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:29:47.644 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:29:47.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:29:47.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:29:47.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:29:47.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:29:47.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:29:47.730 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:29:48.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:29:48.373 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-18T11:29:48.373 INFO:tasks.workunit.client.0.smithi157.stderr:2 2024-09-18T11:29:48.373 INFO:tasks.workunit.client.0.smithi157.stderr:3' 2024-09-18T11:29:48.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:29:48.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:48.373 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:29:48.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836512 2024-09-18T11:29:48.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836512 2024-09-18T11:29:48.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836512' 2024-09-18T11:29:48.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:48.476 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:29:48.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672984 2024-09-18T11:29:48.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672984 2024-09-18T11:29:48.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836512 1-42949672984' 2024-09-18T11:29:48.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:48.584 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:29:48.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542165 2024-09-18T11:29:48.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542165 2024-09-18T11:29:48.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836512 1-42949672984 2-60129542165' 2024-09-18T11:29:48.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:48.732 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:29:48.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378639 2024-09-18T11:29:48.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378639 2024-09-18T11:29:48.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836512 1-42949672984 2-60129542165 3-81604378639' 2024-09-18T11:29:48.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:48.836 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836512 2024-09-18T11:29:48.836 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:48.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:29:48.837 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836512 2024-09-18T11:29:48.837 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:48.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836512 2024-09-18T11:29:48.838 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836512 2024-09-18T11:29:48.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-18T11:29:48.839 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:29:49.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836512 2024-09-18T11:29:49.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:29:50.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:29:50.184 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:29:50.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-18T11:29:50.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:50.533 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672984 2024-09-18T11:29:50.533 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:50.534 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:29:50.534 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672984 2024-09-18T11:29:50.534 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:50.535 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672984 2024-09-18T11:29:50.536 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672984 2024-09-18T11:29:50.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-18T11:29:50.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:29:50.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672985 -lt 42949672984 2024-09-18T11:29:50.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:50.884 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-60129542165 2024-09-18T11:29:50.884 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:50.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:29:50.885 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-60129542165 2024-09-18T11:29:50.885 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:50.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542165 2024-09-18T11:29:50.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 60129542165' 2024-09-18T11:29:50.886 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 60129542165 2024-09-18T11:29:50.886 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:29:51.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542165 -lt 60129542165 2024-09-18T11:29:51.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:51.234 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-81604378639 2024-09-18T11:29:51.234 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:51.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:29:51.235 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-81604378639 2024-09-18T11:29:51.235 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:51.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378639 2024-09-18T11:29:51.236 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.3 seq 81604378639 2024-09-18T11:29:51.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 81604378639' 2024-09-18T11:29:51.237 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:29:51.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378639 -lt 81604378639 2024-09-18T11:29:51.595 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:29:51.595 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:29:51.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:29:52.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-18T11:29:52.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:29:52.071 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:29:52.071 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:29:52.071 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:29:52.071 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:29:52.071 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:29:52.071 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:29:52.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=24 2024-09-18T11:29:52.411 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:29:52.411 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:29:52.412 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:29:52.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 24 = 32 2024-09-18T11:29:52.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 24 '!=' -1 2024-09-18T11:29:52.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T11:29:52.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=24 2024-09-18T11:29:52.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:29:52.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T11:29:52.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:29:52.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:29:52.980 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:29:52.980 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:29:52.980 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:29:52.980 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:29:52.980 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:29:52.981 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:29:53.325 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=24 2024-09-18T11:29:53.325 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:29:53.325 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:29:53.325 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:29:53.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 24 = 32 2024-09-18T11:29:53.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 24 '!=' 24 2024-09-18T11:29:53.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T11:29:53.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T11:29:53.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T11:29:53.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T11:29:53.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T11:29:53.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T11:29:53.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T11:29:54.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T11:29:54.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T11:29:54.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-18T11:29:54.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:29:54.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T11:29:54.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:29:54.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:29:54.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:29:54.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:29:54.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:29:54.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:29:54.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:29:54.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:29:54.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-18T11:29:54.826 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:29:54.826 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:29:54.826 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:29:55.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-18T11:29:55.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:29:55.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:29:55.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-18T11:29:55.627 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:29:55.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-09-18T11:29:55.970 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:29:55.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-09-18T11:29:56.224 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:29:56.225 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-18T11:29:56.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-09-18T11:29:56.464 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:29:56.465 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-18T11:29:56.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-09-18T11:29:56.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2024-09-18T11:29:56.822 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:29:56.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2024-09-18T11:29:56.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2024-09-18T11:29:57.166 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:29:57.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2024-09-18T11:29:57.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2024-09-18T11:29:57.512 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:29:57.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2024-09-18T11:29:57.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2024-09-18T11:29:57.901 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:29:57.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2024-09-18T11:29:57.902 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:29:57.902 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:29:57.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:29:57.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:29:57.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:29:58.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:29:58.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2024-09-18T11:29:58.848 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already down. 2024-09-18T11:29:58.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2024-09-18T11:29:58.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-09-18T11:29:58.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-09-18T11:29:58.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-09-18T11:29:58.859 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-09-18T11:29:58.860 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:29:58.860 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:29:58.860 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:29:58.860 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:29:58.860 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:29:58.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:29:58.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-09-18T11:29:58.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-09-18T11:29:58.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-09-18T11:29:58.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:29:58.948 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:29:59.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:29:59.450 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-18T11:29:59.450 INFO:tasks.workunit.client.0.smithi157.stderr:2 2024-09-18T11:29:59.450 INFO:tasks.workunit.client.0.smithi157.stderr:3' 2024-09-18T11:29:59.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:29:59.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:59.450 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:29:59.521 INFO:tasks.workunit.client.0.smithi157.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-18T11:29:59.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-18T11:29:59.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-18T11:29:59.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-18T11:29:59.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:59.523 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:29:59.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672989 2024-09-18T11:29:59.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672989 2024-09-18T11:29:59.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949672989' 2024-09-18T11:29:59.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:59.628 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:29:59.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542169 2024-09-18T11:29:59.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542169 2024-09-18T11:29:59.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949672989 2-60129542169' 2024-09-18T11:29:59.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:59.732 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:29:59.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378643 2024-09-18T11:29:59.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378643 2024-09-18T11:29:59.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949672989 2-60129542169 3-81604378643' 2024-09-18T11:29:59.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:59.837 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672989 2024-09-18T11:29:59.838 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:59.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:29:59.839 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672989 2024-09-18T11:29:59.839 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:59.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672989 2024-09-18T11:29:59.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-09-18T11:29:59.840 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672989 2024-09-18T11:29:59.840 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:30:00.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672989 -lt 42949672989 2024-09-18T11:30:00.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:00.211 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-60129542169 2024-09-18T11:30:00.211 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:00.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:30:00.212 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-60129542169 2024-09-18T11:30:00.212 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:00.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542169 2024-09-18T11:30:00.213 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 60129542169 2024-09-18T11:30:00.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 60129542169' 2024-09-18T11:30:00.213 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:30:00.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542169 -lt 60129542169 2024-09-18T11:30:00.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:00.581 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-81604378643 2024-09-18T11:30:00.581 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:00.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:30:00.582 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-81604378643 2024-09-18T11:30:00.582 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:00.583 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378643 2024-09-18T11:30:00.583 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 81604378643' 2024-09-18T11:30:00.584 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.3 seq 81604378643 2024-09-18T11:30:00.584 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:30:00.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378643 -lt 81604378643 2024-09-18T11:30:00.982 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-09-18T11:30:00.982 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:30:00.982 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:30:01.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 32 == 0 2024-09-18T11:30:01.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-09-18T11:30:01.451 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-09-18T11:30:01.451 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-09-18T11:30:01.451 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-18T11:30:01.451 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-18T11:30:01.451 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-18T11:30:01.451 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-18T11:30:01.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=27 2024-09-18T11:30:01.791 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-09-18T11:30:01.791 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:30:01.791 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:30:02.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 27 = 32 2024-09-18T11:30:02.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_peered: test 27 '!=' -1 2024-09-18T11:30:02.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_peered: loop=0 2024-09-18T11:30:02.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_peered: num_peered=27 2024-09-18T11:30:02.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_peered: eval 2024-09-18T11:30:02.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1727: wait_for_peered: sleep 0.1 2024-09-18T11:30:02.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1728: wait_for_peered: loop+=1 2024-09-18T11:30:02.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-09-18T11:30:02.379 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-09-18T11:30:02.379 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-09-18T11:30:02.379 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-18T11:30:02.379 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-18T11:30:02.379 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-18T11:30:02.379 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-18T11:30:02.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=32 2024-09-18T11:30:02.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-09-18T11:30:02.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:30:02.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:30:03.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 32 = 32 2024-09-18T11:30:03.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-09-18T11:30:03.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-09-18T11:30:03.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2024-09-18T11:30:03.524 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2024-09-18T11:30:03.535 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-18T11:30:03.879 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:30:03.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-09-18T11:30:04.128 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:30:04.128 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-18T11:30:04.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-09-18T11:30:04.370 INFO:tasks.workunit.client.0.smithi157.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-18T11:30:04.370 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-18T11:30:04.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-18T11:30:04.374 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:04.374 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:04.375 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:04.375 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:04.375 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:04.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:04.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-18T11:30:04.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-18T11:30:04.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:30:04.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-18T11:30:04.483 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:04.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:04.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:04.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:04.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:04.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:04.489 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:30:04.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:30:04.490 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:30:04.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:30:04.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:30:04.491 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:30:04.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:30:04.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:30:04.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:30:04.493 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:30:04.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:30:04.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:30:04.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-18T11:30:04.517 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:30:04.518 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:04.518 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:04.518 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53871 2024-09-18T11:30:04.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:30:04.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:30:04.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-18T11:30:04.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-18T11:30:04.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-18T11:30:04.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-18T11:30:04.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-18T11:30:04.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:30:04.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-18T11:30:04.520 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:04.520 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:04.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:04.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:04.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:04.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:04.521 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:30:04.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:30:04.522 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:30:04.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:30:04.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:30:04.523 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:30:04.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:30:04.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:30:04.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:30:04.525 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:30:04.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:30:04.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:30:04.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-18T11:30:04.527 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:30:04.528 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:04.528 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:04.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53871 2024-09-18T11:30:04.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:30:04.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:30:04.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-18T11:30:04.529 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:30:04.529 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:04.530 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:04.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53871 2024-09-18T11:30:04.532 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:30:04.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:30:04.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:30:04.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-18T11:30:04.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2024-09-18T11:30:04.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2024-09-18T11:30:04.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-18T11:30:04.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-18T11:30:04.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:30:04.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:30:04.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:30:04.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:30:04.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-18T11:30:04.534 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-18T11:30:04.891 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:30:04.892 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:30:04.892 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:30:04.892 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:30:04.892 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:04.892 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:04.892 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:30:04.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-18T11:30:05.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:30:05.144 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:30:05.144 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:05.144 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:30:05.144 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:30:05.144 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:30:05.144 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:30:05.144 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:30:05.145 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:30:05.145 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:05.145 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:05.145 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:05.146 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.a.asok 2024-09-18T11:30:05.146 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:05.146 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.a.asok config get fsid 2024-09-18T11:30:05.222 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:30:05.222 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:05.222 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:30:05.223 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:30:05.223 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:30:05.223 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:30:05.223 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:30:05.223 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:30:05.223 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:05.223 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:05.223 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:05.223 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.a.asok 2024-09-18T11:30:05.223 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:05.223 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.a.asok config get mon_host 2024-09-18T11:30:05.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2024-09-18T11:30:06.069 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (a) after stopping [dne] 2024-09-18T11:30:06.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2024-09-18T11:30:06.232 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a] 2024-09-18T11:30:06.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2024-09-18T11:30:06.369 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-18T11:30:06.371 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2024-09-18T11:30:06.519 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: removing mon.a would break quorum 2024-09-18T11:30:06.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2024-09-18T11:30:06.707 INFO:tasks.workunit.client.0.smithi157.stderr:mon.dne does not exist 2024-09-18T11:30:06.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-18T11:30:06.717 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:06.717 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:06.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:06.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:06.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:06.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:06.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-18T11:30:06.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-18T11:30:06.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:30:06.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-18T11:30:06.823 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:06.823 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:06.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:06.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:06.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:06.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:06.826 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:30:06.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:30:06.827 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:30:06.828 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:30:06.828 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:30:06.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:30:06.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:30:06.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:30:06.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:30:06.830 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:30:06.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:30:06.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:30:06.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-18T11:30:06.833 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:30:06.833 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:06.833 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:06.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53871 2024-09-18T11:30:06.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:30:06.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:30:06.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-18T11:30:06.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-18T11:30:06.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-18T11:30:06.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-18T11:30:06.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-18T11:30:06.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:30:06.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-18T11:30:06.835 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:06.835 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:06.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:06.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:06.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:06.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:06.836 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:30:06.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:30:06.837 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:30:06.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:30:06.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:30:06.838 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:30:06.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:30:06.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:30:06.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:30:06.840 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:30:06.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:30:06.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:30:06.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-18T11:30:06.842 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:30:06.842 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:06.843 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:06.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53871 2024-09-18T11:30:06.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:30:06.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:30:06.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-18T11:30:06.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:30:06.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:06.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:06.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53871 2024-09-18T11:30:06.847 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:30:06.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:30:06.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:30:06.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-18T11:30:06.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2024-09-18T11:30:06.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2024-09-18T11:30:06.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2024-09-18T11:30:06.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-18T11:30:06.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:30:06.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:30:06.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:30:06.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:30:06.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-18T11:30:06.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-18T11:30:07.268 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:30:07.268 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:30:07.268 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:30:07.269 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:30:07.269 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:07.269 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:07.269 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:30:07.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-18T11:30:07.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:30:07.487 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:30:07.487 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:07.487 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:30:07.487 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:30:07.487 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:30:07.487 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:30:07.487 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:30:07.487 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:30:07.488 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:07.488 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:07.488 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:07.488 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.a.asok 2024-09-18T11:30:07.488 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:07.488 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.a.asok config get fsid 2024-09-18T11:30:07.564 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:30:07.564 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:07.564 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:30:07.564 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:30:07.564 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:30:07.564 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:30:07.565 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:30:07.565 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:30:07.565 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:07.565 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:07.565 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:07.565 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.a.asok 2024-09-18T11:30:07.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:07.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.a.asok config get mon_host 2024-09-18T11:30:07.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-18T11:30:07.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:30:07.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:30:07.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-18T11:30:07.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:30:07.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-09-18T11:30:07.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-18T11:30:08.030 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:30:08.030 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:30:08.030 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:30:08.031 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:30:08.031 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:08.031 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:08.031 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:30:08.031 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-18T11:30:08.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:30:08.299 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-18T11:30:08.299 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:08.299 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-18T11:30:08.299 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:30:08.299 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:30:08.299 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-18T11:30:08.299 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-18T11:30:08.300 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-18T11:30:08.300 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:08.300 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:08.300 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:08.300 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.b.asok 2024-09-18T11:30:08.300 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:08.300 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.b.asok config get fsid 2024-09-18T11:30:08.373 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-18T11:30:08.374 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:08.374 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-18T11:30:08.374 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:30:08.374 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:30:08.374 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-18T11:30:08.374 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-18T11:30:08.374 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-18T11:30:08.374 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:08.374 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:08.374 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:08.374 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.b.asok 2024-09-18T11:30:08.375 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:08.375 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.b.asok config get mon_host 2024-09-18T11:30:08.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2024-09-18T11:30:11.627 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (a,b) after stopping [dne] 2024-09-18T11:30:11.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2024-09-18T11:30:11.826 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons [a] 2024-09-18T11:30:11.828 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2024-09-18T11:30:12.061 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b] 2024-09-18T11:30:12.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2024-09-18T11:30:12.197 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b] 2024-09-18T11:30:12.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2024-09-18T11:30:12.368 INFO:tasks.workunit.client.0.smithi157.stderr:adding another mon that is not yet online will not break quorum 2024-09-18T11:30:12.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2024-09-18T11:30:12.551 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.a 2024-09-18T11:30:12.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2024-09-18T11:30:12.815 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.b 2024-09-18T11:30:12.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2024-09-18T11:30:13.004 INFO:tasks.workunit.client.0.smithi157.stderr:mon.dne does not exist 2024-09-18T11:30:13.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-18T11:30:13.014 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:13.014 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:13.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:13.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:13.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:13.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:13.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-18T11:30:13.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-18T11:30:13.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:30:13.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-18T11:30:13.120 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:13.120 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:13.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:13.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:13.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:13.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:13.124 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:30:13.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:30:13.125 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:30:13.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:30:13.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:30:13.126 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:30:13.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:30:13.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:30:13.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:30:13.127 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:30:13.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:30:13.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:30:13.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-18T11:30:13.131 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:30:13.131 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:13.131 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:13.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53871 2024-09-18T11:30:13.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:30:13.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:30:13.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-18T11:30:13.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-18T11:30:13.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-18T11:30:13.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-18T11:30:13.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-18T11:30:13.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:30:13.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-18T11:30:13.134 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:13.134 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:13.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:13.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:13.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:13.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:13.135 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:30:13.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:30:13.135 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:30:13.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:30:13.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:30:13.137 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:30:13.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:30:13.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:30:13.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:30:13.138 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:30:13.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:30:13.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:30:13.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-18T11:30:13.140 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:30:13.141 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:13.141 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:13.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53871 2024-09-18T11:30:13.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:30:13.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:30:13.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-18T11:30:13.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:30:13.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:13.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:13.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53871 2024-09-18T11:30:13.145 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:30:13.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:30:13.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:30:13.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-18T11:30:13.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2024-09-18T11:30:13.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2024-09-18T11:30:13.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2024-09-18T11:30:13.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-18T11:30:13.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:30:13.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:30:13.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:30:13.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:30:13.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-18T11:30:13.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-18T11:30:13.506 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:30:13.506 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:30:13.506 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:30:13.507 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:30:13.507 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:13.507 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:13.507 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:30:13.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-18T11:30:13.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:30:13.742 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:30:13.742 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:13.742 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:30:13.742 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:30:13.742 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:30:13.742 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:30:13.742 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:30:13.742 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:30:13.743 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:13.743 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:13.743 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:13.744 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.a.asok 2024-09-18T11:30:13.744 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:13.744 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.a.asok config get fsid 2024-09-18T11:30:13.819 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:30:13.819 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:13.819 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:30:13.819 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:30:13.819 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:30:13.819 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:30:13.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:30:13.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:30:13.820 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:13.820 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:13.820 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:13.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.a.asok 2024-09-18T11:30:13.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:13.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.a.asok config get mon_host 2024-09-18T11:30:13.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-18T11:30:13.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:30:13.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:30:13.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-18T11:30:13.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:30:13.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-09-18T11:30:13.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-18T11:30:14.234 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:30:14.234 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:30:14.234 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:30:14.234 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:30:14.235 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:14.235 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:14.235 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:30:14.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-18T11:30:14.480 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:30:14.481 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-18T11:30:14.481 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:14.482 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-18T11:30:14.482 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:30:14.482 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:30:14.482 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-18T11:30:14.482 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-18T11:30:14.482 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-18T11:30:14.482 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:14.482 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:14.482 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:14.482 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.b.asok 2024-09-18T11:30:14.482 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:14.482 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.b.asok config get fsid 2024-09-18T11:30:14.558 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-18T11:30:14.558 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:14.558 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-18T11:30:14.558 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:30:14.558 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:30:14.559 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-18T11:30:14.559 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-18T11:30:14.559 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-18T11:30:14.559 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:14.559 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:14.559 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:14.559 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.b.asok 2024-09-18T11:30:14.560 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:14.560 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.b.asok config get mon_host 2024-09-18T11:30:14.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-18T11:30:14.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:30:14.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:30:14.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-09-18T11:30:14.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:30:14.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-09-18T11:30:14.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-18T11:30:14.990 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:30:14.990 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:30:14.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:30:14.991 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:30:14.991 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:14.991 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:14.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:30:14.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-18T11:30:15.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:30:15.276 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-09-18T11:30:15.276 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:15.276 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-18T11:30:15.276 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:30:15.276 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:30:15.276 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-18T11:30:15.276 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-18T11:30:15.276 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-18T11:30:15.277 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:15.277 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:15.277 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:15.277 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.c.asok 2024-09-18T11:30:15.277 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:15.278 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.c.asok config get fsid 2024-09-18T11:30:15.353 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-09-18T11:30:15.353 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:15.353 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-18T11:30:15.353 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:30:15.353 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:30:15.353 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-18T11:30:15.353 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-18T11:30:15.353 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-18T11:30:15.353 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:15.353 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:15.353 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:15.354 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.c.asok 2024-09-18T11:30:15.354 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:15.354 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.c.asok config get mon_host 2024-09-18T11:30:15.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2024-09-18T11:30:15.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-18T11:30:15.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-09-18T11:30:15.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-18T11:30:15.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-09-18T11:30:15.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-18T11:30:15.427 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-18T11:30:15.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726659075 2024-09-18T11:30:15.428 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:15.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659015 -lt 1726659075 ]] 2024-09-18T11:30:15.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:30:15.429 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:18.615 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:18.615 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:18.615 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:18.616 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:30:18.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-18T11:30:18.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-18T11:30:18.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-18T11:30:18.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-18T11:30:18.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-18T11:30:18.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2024-09-18T11:30:18.829 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (a,b,c) after stopping [dne] 2024-09-18T11:30:18.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-09-18T11:30:19.077 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (b,c) after stopping [a] 2024-09-18T11:30:19.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-09-18T11:30:19.298 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (a,c) after stopping [b] 2024-09-18T11:30:19.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-09-18T11:30:19.609 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (a,b) after stopping [c] 2024-09-18T11:30:19.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2024-09-18T11:30:19.893 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a,b] 2024-09-18T11:30:19.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2024-09-18T11:30:20.126 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b,c] 2024-09-18T11:30:20.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2024-09-18T11:30:20.312 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b,c] 2024-09-18T11:30:20.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-09-18T11:30:20.658 INFO:tasks.workunit.client.0.smithi157.stderr:adding another mon that is not yet online will not break quorum 2024-09-18T11:30:20.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-09-18T11:30:20.939 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.a 2024-09-18T11:30:20.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-09-18T11:30:21.207 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.b 2024-09-18T11:30:21.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-09-18T11:30:21.458 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.c 2024-09-18T11:30:21.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2024-09-18T11:30:21.468 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:21.468 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:21.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:21.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:21.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:21.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:21.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2024-09-18T11:30:21.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-18T11:30:21.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=2 2024-09-18T11:30:21.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-18T11:30:21.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 2 ]] 2024-09-18T11:30:21.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-18T11:30:21.574 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-18T11:30:21.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726659081 2024-09-18T11:30:21.574 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:21.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659021 -lt 1726659081 ]] 2024-09-18T11:30:21.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-18T11:30:21.575 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:21.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:21.754 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:21.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-18T11:30:21.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:21.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:21.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:21.757 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:21.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659021 -lt 1726659081 ]] 2024-09-18T11:30:21.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-18T11:30:21.758 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:21.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:21.938 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:21.938 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:21.938 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-18T11:30:21.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:21.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:21.941 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:21.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659021 -lt 1726659081 ]] 2024-09-18T11:30:21.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-18T11:30:21.942 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:22.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:22.119 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:22.119 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:22.120 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-18T11:30:22.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:22.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:22.122 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:22.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659022 -lt 1726659081 ]] 2024-09-18T11:30:22.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-18T11:30:22.123 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:22.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:22.302 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:22.302 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:22.302 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-18T11:30:22.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:22.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:22.305 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:22.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659022 -lt 1726659081 ]] 2024-09-18T11:30:22.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-18T11:30:22.306 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:22.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:22.484 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:22.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:22.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-18T11:30:22.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:22.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:22.487 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:22.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659022 -lt 1726659081 ]] 2024-09-18T11:30:22.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-18T11:30:22.488 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:25.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:25.674 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:25.674 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-18T11:30:25.674 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:25.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:25.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:25.677 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:25.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659025 -lt 1726659081 ]] 2024-09-18T11:30:25.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-18T11:30:25.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:25.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:25.858 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:25.859 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-18T11:30:25.859 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:25.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:25.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:25.862 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:25.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659025 -lt 1726659081 ]] 2024-09-18T11:30:25.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-18T11:30:25.863 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:29.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:29.049 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:29.049 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:29.049 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-18T11:30:29.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:29.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:29.052 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:29.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659029 -lt 1726659081 ]] 2024-09-18T11:30:29.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-18T11:30:29.053 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:38.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:38.240 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:38.241 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:13.160443Z","created":"2024-09-18T11:30:13.160443Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:38.241 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-18T11:30:38.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-18T11:30:38.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-18T11:30:38.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-18T11:30:38.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-18T11:30:38.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-18T11:30:38.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-09-18T11:30:41.382 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a] 2024-09-18T11:30:41.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-09-18T11:30:41.664 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (a,c) after stopping [b] 2024-09-18T11:30:41.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-09-18T11:30:41.840 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [c] 2024-09-18T11:30:41.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-09-18T11:30:44.979 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-18T11:30:44.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-09-18T11:30:45.181 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: removing mon.a would break quorum 2024-09-18T11:30:45.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-09-18T11:30:45.384 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.b 2024-09-18T11:30:45.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-09-18T11:30:45.563 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: removing mon.c would break quorum 2024-09-18T11:30:45.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-18T11:30:45.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:45.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:45.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:45.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:45.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:45.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:45.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-18T11:30:45.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-18T11:30:45.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:30:45.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-18T11:30:45.672 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:45.672 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:45.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:45.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:45.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:45.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:45.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:30:45.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:30:45.677 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:30:45.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:30:45.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:30:45.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:30:45.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:30:45.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:30:45.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:30:45.679 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:30:45.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:30:45.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:30:45.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-18T11:30:45.685 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:30:45.685 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:45.685 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:45.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53871 2024-09-18T11:30:45.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:30:45.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:30:45.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-18T11:30:45.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-18T11:30:45.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-18T11:30:45.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-18T11:30:45.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-18T11:30:45.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:30:45.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-18T11:30:45.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:45.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:45.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:45.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:45.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:45.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:45.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:30:45.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:30:45.689 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:30:45.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:30:45.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:30:45.690 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:30:45.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:30:45.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:30:45.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:30:45.692 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:30:45.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:30:45.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:30:45.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-18T11:30:45.694 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:30:45.695 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:45.695 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:45.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53871 2024-09-18T11:30:45.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:30:45.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:30:45.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-18T11:30:45.697 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:30:45.697 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:45.697 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:45.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53871 2024-09-18T11:30:45.699 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:30:45.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:30:45.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:30:45.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-18T11:30:45.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2024-09-18T11:30:45.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2024-09-18T11:30:45.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2024-09-18T11:30:45.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-18T11:30:45.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:30:45.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:30:45.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:30:45.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:30:45.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-18T11:30:45.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-18T11:30:46.061 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:30:46.061 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:30:46.061 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:30:46.061 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:30:46.061 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:46.061 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:46.061 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:30:46.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-18T11:30:46.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:30:46.313 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:30:46.313 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:46.313 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:30:46.313 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:30:46.313 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:30:46.314 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:30:46.314 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:30:46.314 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:30:46.314 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:46.314 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:46.314 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:46.314 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.a.asok 2024-09-18T11:30:46.314 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:46.315 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.a.asok config get fsid 2024-09-18T11:30:46.388 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:30:46.388 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:46.388 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:30:46.388 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:30:46.388 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:30:46.388 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:30:46.388 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:30:46.389 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:30:46.389 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:46.389 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:46.389 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:46.389 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.a.asok 2024-09-18T11:30:46.389 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:46.389 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.a.asok config get mon_host 2024-09-18T11:30:46.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-18T11:30:46.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:30:46.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:30:46.463 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-18T11:30:46.463 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:30:46.463 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-09-18T11:30:46.463 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-18T11:30:46.805 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:30:46.806 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:30:46.806 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:30:46.806 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:30:46.806 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:46.806 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:46.806 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:30:46.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-18T11:30:47.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:30:47.066 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-18T11:30:47.066 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:47.066 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-18T11:30:47.066 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:30:47.066 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:30:47.066 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-18T11:30:47.067 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-18T11:30:47.067 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-18T11:30:47.067 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:47.068 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:47.068 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:47.068 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.b.asok 2024-09-18T11:30:47.068 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:47.068 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.b.asok config get fsid 2024-09-18T11:30:47.140 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-18T11:30:47.140 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:47.141 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-18T11:30:47.141 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:30:47.141 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:30:47.141 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-18T11:30:47.141 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-18T11:30:47.141 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-18T11:30:47.141 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:47.141 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:47.141 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:47.141 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.b.asok 2024-09-18T11:30:47.141 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:47.141 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.b.asok config get mon_host 2024-09-18T11:30:47.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-18T11:30:47.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:30:47.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:30:47.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-09-18T11:30:47.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:30:47.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-09-18T11:30:47.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-18T11:30:47.560 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:30:47.560 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:30:47.560 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:30:47.560 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:30:47.560 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:47.560 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:47.560 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:30:47.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-18T11:30:47.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:30:47.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-09-18T11:30:47.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:47.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-18T11:30:47.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:30:47.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:30:47.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-18T11:30:47.821 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-18T11:30:47.821 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-18T11:30:47.821 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:47.821 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:47.822 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:47.822 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.c.asok 2024-09-18T11:30:47.822 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:47.822 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.c.asok config get fsid 2024-09-18T11:30:47.895 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-09-18T11:30:47.895 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:47.895 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-18T11:30:47.895 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:30:47.895 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:30:47.895 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-18T11:30:47.895 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-18T11:30:47.895 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-18T11:30:47.895 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:47.895 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:47.896 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:47.896 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.c.asok 2024-09-18T11:30:47.896 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:47.896 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.c.asok config get mon_host 2024-09-18T11:30:47.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-09-18T11:30:47.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:30:47.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:30:47.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-09-18T11:30:47.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:30:47.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-09-18T11:30:47.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-09-18T11:30:48.305 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:30:48.305 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:30:48.305 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:30:48.305 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:30:48.305 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:48.305 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:48.305 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:30:48.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-09-18T11:30:48.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:30:48.591 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-09-18T11:30:48.591 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:48.591 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-09-18T11:30:48.591 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:30:48.591 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:30:48.591 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-09-18T11:30:48.591 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-09-18T11:30:48.591 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-09-18T11:30:48.592 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:48.592 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:48.592 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:48.592 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.d.asok 2024-09-18T11:30:48.593 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:48.593 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.d.asok config get fsid 2024-09-18T11:30:48.667 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-09-18T11:30:48.667 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:30:48.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-09-18T11:30:48.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:30:48.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:30:48.668 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-09-18T11:30:48.668 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-09-18T11:30:48.668 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-09-18T11:30:48.668 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:30:48.668 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:48.668 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:30:48.668 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.d.asok 2024-09-18T11:30:48.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:30:48.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.d.asok config get mon_host 2024-09-18T11:30:48.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2024-09-18T11:30:48.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-18T11:30:48.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-09-18T11:30:48.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-18T11:30:48.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-09-18T11:30:48.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-18T11:30:48.740 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-18T11:30:48.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726659108 2024-09-18T11:30:48.740 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:48.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659048 -lt 1726659108 ]] 2024-09-18T11:30:48.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-18T11:30:48.741 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:51.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:51.931 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:51.931 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-18T11:30:51.931 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:51.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-18T11:30:51.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-18T11:30:51.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-18T11:30:51.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-18T11:30:51.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-18T11:30:51.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2024-09-18T11:30:52.186 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (a,b,c,d) after stopping [dne] 2024-09-18T11:30:52.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-09-18T11:30:52.496 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-09-18T11:30:52.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-09-18T11:30:52.771 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (a,c,d) after stopping [b] 2024-09-18T11:30:52.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-09-18T11:30:53.153 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (a,b,d) after stopping [c] 2024-09-18T11:30:53.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-09-18T11:30:53.425 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (a,b,c) after stopping [d] 2024-09-18T11:30:53.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2024-09-18T11:30:53.663 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [a,b] 2024-09-18T11:30:53.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2024-09-18T11:30:53.973 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons [c,d] 2024-09-18T11:30:53.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-09-18T11:30:54.275 INFO:tasks.workunit.client.0.smithi157.stderr:adding another mon that is not yet online will not break quorum 2024-09-18T11:30:54.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-09-18T11:30:54.589 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.a 2024-09-18T11:30:54.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-09-18T11:30:54.842 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.b 2024-09-18T11:30:54.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-09-18T11:30:55.110 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.c 2024-09-18T11:30:55.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-09-18T11:30:55.120 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:55.120 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:55.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:55.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:55.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:55.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:55.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2024-09-18T11:30:55.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-18T11:30:55.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-09-18T11:30:55.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-18T11:30:55.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-09-18T11:30:55.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-18T11:30:55.226 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-18T11:30:55.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726659115 2024-09-18T11:30:55.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:55.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659055 -lt 1726659115 ]] 2024-09-18T11:30:55.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:30:55.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:55.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:55.407 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:55.407 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:30:55.407 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:55.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:55.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:55.410 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:55.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659055 -lt 1726659115 ]] 2024-09-18T11:30:55.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:30:55.411 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:55.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:55.590 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:55.590 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:30:55.590 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:55.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:55.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:55.593 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:55.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659055 -lt 1726659115 ]] 2024-09-18T11:30:55.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:30:55.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:55.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:55.772 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:55.772 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:55.772 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:30:55.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:55.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:55.775 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:55.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659055 -lt 1726659115 ]] 2024-09-18T11:30:55.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:30:55.776 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:55.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:55.956 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:55.956 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:55.956 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:30:55.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:55.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:55.959 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:55.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659055 -lt 1726659115 ]] 2024-09-18T11:30:55.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:30:55.960 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:56.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:56.140 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:56.140 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:56.140 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:30:56.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:56.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:56.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:56.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659056 -lt 1726659115 ]] 2024-09-18T11:30:56.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:30:56.144 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:56.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:56.322 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:56.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:56.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:30:56.325 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:56.325 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:56.326 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:56.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659056 -lt 1726659115 ]] 2024-09-18T11:30:56.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:30:56.327 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:56.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:56.504 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:56.505 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:56.505 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:30:56.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:56.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:56.508 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:56.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659056 -lt 1726659115 ]] 2024-09-18T11:30:56.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:30:56.509 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:56.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:56.688 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:56.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:30:56.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:56.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:56.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:56.691 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:56.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659056 -lt 1726659115 ]] 2024-09-18T11:30:56.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:30:56.692 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:30:56.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:30:56.871 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:56.871 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:30:56.871 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:30:56.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:30:56.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:30:56.874 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:30:56.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659056 -lt 1726659115 ]] 2024-09-18T11:30:56.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:30:56.875 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:00.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:00.061 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:00.062 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:31:00.062 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:00.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:00.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:00.065 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:00.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659060 -lt 1726659115 ]] 2024-09-18T11:31:00.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:31:00.066 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:00.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:00.246 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:00.247 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:31:00.247 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:00.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:00.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:00.250 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:00.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659060 -lt 1726659115 ]] 2024-09-18T11:31:00.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:31:00.251 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:00.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:00.431 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:00.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:00.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:31:00.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:00.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:00.434 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:00.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659060 -lt 1726659115 ]] 2024-09-18T11:31:00.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:31:00.436 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:03.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:03.623 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:03.624 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:03.624 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:31:03.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:03.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:03.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:03.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659063 -lt 1726659115 ]] 2024-09-18T11:31:03.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:31:03.628 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:03.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:03.807 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:03.807 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:03.807 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:31:03.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:03.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:03.810 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:03.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659063 -lt 1726659115 ]] 2024-09-18T11:31:03.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:31:03.811 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:03.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:03.991 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:03.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:03.992 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:31:03.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:03.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:03.997 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:03.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659063 -lt 1726659115 ]] 2024-09-18T11:31:03.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:31:03.997 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:10.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:10.394 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":6,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:10.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:30:45.714169Z","created":"2024-09-18T11:30:45.714169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:10.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:31:10.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-18T11:31:10.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-18T11:31:10.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-18T11:31:10.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-18T11:31:10.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-18T11:31:10.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-09-18T11:31:10.773 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-09-18T11:31:10.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-09-18T11:31:11.016 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-09-18T11:31:11.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-09-18T11:31:11.225 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-09-18T11:31:11.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-09-18T11:31:14.367 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-09-18T11:31:14.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-09-18T11:31:14.650 INFO:tasks.workunit.client.0.smithi157.stderr:adding another mon that is not yet online will not break quorum 2024-09-18T11:31:14.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-09-18T11:31:14.851 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.a 2024-09-18T11:31:14.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-09-18T11:31:15.085 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.b 2024-09-18T11:31:15.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-09-18T11:31:18.273 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.c 2024-09-18T11:31:18.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2024-09-18T11:31:18.520 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.d 2024-09-18T11:31:18.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-18T11:31:18.530 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:31:18.530 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:31:18.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:31:18.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:31:18.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:31:18.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:31:18.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-18T11:31:18.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-18T11:31:18.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:31:18.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-18T11:31:18.638 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:31:18.638 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:31:18.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:31:18.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:31:18.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:31:18.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:31:18.642 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:31:18.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:31:18.643 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:31:18.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:31:18.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:31:18.644 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:31:18.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:31:18.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:31:18.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:31:18.645 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:31:18.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:31:18.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:31:18.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-18T11:31:18.652 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:31:18.653 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:18.653 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:18.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53871 2024-09-18T11:31:18.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:31:18.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:31:18.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-18T11:31:18.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-18T11:31:18.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-18T11:31:18.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-18T11:31:18.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-18T11:31:18.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:31:18.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-18T11:31:18.655 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:31:18.655 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:31:18.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:31:18.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:31:18.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:31:18.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:31:18.656 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:31:18.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:31:18.657 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:31:18.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:31:18.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:31:18.658 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:31:18.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:31:18.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:31:18.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:31:18.660 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:31:18.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:31:18.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:31:18.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-18T11:31:18.662 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:31:18.662 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:18.662 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:18.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53871 2024-09-18T11:31:18.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:31:18.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:31:18.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-18T11:31:18.664 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:31:18.664 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:18.665 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:18.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53871 2024-09-18T11:31:18.667 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:31:18.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:31:18.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:31:18.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-18T11:31:18.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2024-09-18T11:31:18.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2024-09-18T11:31:18.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=5be5d46b-e374-428a-af2b-68c67ab24ab2 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2024-09-18T11:31:18.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-18T11:31:18.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:31:18.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:31:18.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:31:18.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:31:18.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-18T11:31:18.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-18T11:31:19.041 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:31:19.041 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:31:19.041 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:31:19.041 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:31:19.041 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:19.041 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:19.041 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:31:19.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-18T11:31:19.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:31:19.292 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:31:19.292 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:31:19.292 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:31:19.292 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:31:19.292 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:31:19.292 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:31:19.293 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:31:19.293 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:31:19.293 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:31:19.293 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:19.293 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:19.293 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.a.asok 2024-09-18T11:31:19.293 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:31:19.293 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.a.asok config get fsid 2024-09-18T11:31:19.367 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:31:19.367 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:31:19.367 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:31:19.367 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:31:19.368 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:31:19.368 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:31:19.368 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:31:19.368 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:31:19.368 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:31:19.368 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:19.368 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:19.368 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.a.asok 2024-09-18T11:31:19.368 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:31:19.368 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.a.asok config get mon_host 2024-09-18T11:31:19.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-18T11:31:19.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:31:19.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:31:19.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-18T11:31:19.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:31:19.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-09-18T11:31:19.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-18T11:31:19.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:31:19.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:31:19.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:31:19.792 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:31:19.793 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:19.793 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:19.793 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:31:19.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-18T11:31:20.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:31:20.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-18T11:31:20.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:31:20.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-18T11:31:20.045 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:31:20.045 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:31:20.045 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-18T11:31:20.045 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-18T11:31:20.045 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-18T11:31:20.046 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:31:20.046 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:20.046 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:20.046 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.b.asok 2024-09-18T11:31:20.046 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:31:20.046 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.b.asok config get fsid 2024-09-18T11:31:20.120 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-18T11:31:20.120 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:31:20.120 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-18T11:31:20.120 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:31:20.120 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:31:20.120 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-18T11:31:20.120 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-18T11:31:20.120 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-18T11:31:20.120 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:31:20.120 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:20.120 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:20.121 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.b.asok 2024-09-18T11:31:20.121 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:31:20.121 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.b.asok config get mon_host 2024-09-18T11:31:20.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-18T11:31:20.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:31:20.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:31:20.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-09-18T11:31:20.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:31:20.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-09-18T11:31:20.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-18T11:31:20.554 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:31:20.554 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:31:20.555 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:31:20.555 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:31:20.555 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:20.555 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:20.555 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:31:20.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-18T11:31:20.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:31:20.816 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-09-18T11:31:20.816 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:31:20.816 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-18T11:31:20.816 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:31:20.816 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:31:20.816 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-18T11:31:20.816 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-18T11:31:20.816 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-18T11:31:20.816 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:31:20.816 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:20.816 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:20.817 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.c.asok 2024-09-18T11:31:20.817 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:31:20.817 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.c.asok config get fsid 2024-09-18T11:31:20.891 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-09-18T11:31:20.891 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:31:20.891 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-18T11:31:20.891 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:31:20.891 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:31:20.892 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-18T11:31:20.892 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-18T11:31:20.892 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-18T11:31:20.892 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:31:20.892 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:20.892 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:20.892 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.c.asok 2024-09-18T11:31:20.892 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:31:20.892 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.c.asok config get mon_host 2024-09-18T11:31:20.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-09-18T11:31:20.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:31:20.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:31:20.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-09-18T11:31:20.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:31:20.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-09-18T11:31:20.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-09-18T11:31:21.316 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:31:21.316 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:31:21.316 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:31:21.316 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:31:21.316 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:21.316 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:21.317 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:31:21.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-09-18T11:31:21.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:31:21.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-09-18T11:31:21.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:31:21.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-09-18T11:31:21.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:31:21.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:31:21.594 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-09-18T11:31:21.595 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-09-18T11:31:21.595 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-09-18T11:31:21.595 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:31:21.595 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:21.595 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:21.595 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.d.asok 2024-09-18T11:31:21.595 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:31:21.595 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.d.asok config get fsid 2024-09-18T11:31:21.669 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-09-18T11:31:21.669 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:31:21.669 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-09-18T11:31:21.669 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:31:21.669 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:31:21.669 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-09-18T11:31:21.669 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-09-18T11:31:21.670 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-09-18T11:31:21.670 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:31:21.670 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:21.670 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:21.670 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.d.asok 2024-09-18T11:31:21.670 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:31:21.670 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.d.asok config get mon_host 2024-09-18T11:31:21.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2024-09-18T11:31:21.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-18T11:31:21.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:31:21.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=e 2024-09-18T11:31:21.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:31:21.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/e 2024-09-18T11:31:21.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2024-09-18T11:31:22.120 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:31:22.120 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:31:22.120 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:31:22.120 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:31:22.120 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:22.120 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:22.120 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53871/$cluster-$name.asok' 2024-09-18T11:31:22.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53871/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2024-09-18T11:31:22.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:31:22.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e fsid 2024-09-18T11:31:22.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:31:22.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-09-18T11:31:22.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:31:22.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:31:22.449 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-09-18T11:31:22.449 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-09-18T11:31:22.449 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-09-18T11:31:22.449 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:31:22.450 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:22.450 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:22.450 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.e.asok 2024-09-18T11:31:22.450 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:31:22.450 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.e.asok config get fsid 2024-09-18T11:31:22.524 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e mon_host 2024-09-18T11:31:22.524 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:31:22.524 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-09-18T11:31:22.524 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:31:22.524 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:31:22.524 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-09-18T11:31:22.524 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-09-18T11:31:22.524 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-09-18T11:31:22.524 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:31:22.524 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:22.524 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:31:22.524 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53871/ceph-mon.e.asok 2024-09-18T11:31:22.525 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:31:22.525 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53871/ceph-mon.e.asok config get mon_host 2024-09-18T11:31:22.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2024-09-18T11:31:22.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-18T11:31:22.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=5 2024-09-18T11:31:22.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-18T11:31:22.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 5 ]] 2024-09-18T11:31:22.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-18T11:31:22.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-18T11:31:22.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726659142 2024-09-18T11:31:22.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:22.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659082 -lt 1726659142 ]] 2024-09-18T11:31:22.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 5' 2024-09-18T11:31:22.600 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:27.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:27.371 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:27.371 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 5' 2024-09-18T11:31:27.371 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:27.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-18T11:31:27.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-18T11:31:27.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-18T11:31:27.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-18T11:31:27.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-18T11:31:27.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2024-09-18T11:31:27.670 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (a,b,c,d,e) after stopping [dne] 2024-09-18T11:31:27.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-18T11:31:27.981 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-09-18T11:31:27.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-18T11:31:28.217 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (a,c,d,e) after stopping [b] 2024-09-18T11:31:28.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-18T11:31:28.483 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (a,b,d,e) after stopping [c] 2024-09-18T11:31:28.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-18T11:31:28.757 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (a,b,c,e) after stopping [d] 2024-09-18T11:31:28.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-18T11:31:29.110 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (a,b,c,d) after stopping [e] 2024-09-18T11:31:29.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2024-09-18T11:31:29.323 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (c,d,e) after stopping [a,b] 2024-09-18T11:31:29.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2024-09-18T11:31:29.590 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (a,b,e) after stopping [c,d] 2024-09-18T11:31:29.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2024-09-18T11:31:29.849 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons [a,b,c] 2024-09-18T11:31:29.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-18T11:31:30.096 INFO:tasks.workunit.client.0.smithi157.stderr:adding another mon that is not yet online will not break quorum 2024-09-18T11:31:30.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-18T11:31:30.311 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.a 2024-09-18T11:31:30.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-18T11:31:30.676 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.b 2024-09-18T11:31:30.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-18T11:31:30.983 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.c 2024-09-18T11:31:30.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-18T11:31:31.288 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.d 2024-09-18T11:31:31.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-18T11:31:31.630 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.e 2024-09-18T11:31:31.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-09-18T11:31:31.640 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:31:31.640 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:31:31.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:31:31.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:31:31.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:31:31.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:31:31.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2024-09-18T11:31:31.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-18T11:31:31.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-09-18T11:31:31.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-18T11:31:31.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-09-18T11:31:31.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-18T11:31:31.746 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-18T11:31:31.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726659151 2024-09-18T11:31:31.746 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:31.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659091 -lt 1726659151 ]] 2024-09-18T11:31:31.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-18T11:31:31.747 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:31.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:31.929 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:31.929 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:31.929 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-18T11:31:31.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:31.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:31.932 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:31.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659091 -lt 1726659151 ]] 2024-09-18T11:31:31.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-18T11:31:31.933 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:32.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:32.114 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:32.114 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:32.114 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-18T11:31:32.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:32.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:32.117 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:32.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659092 -lt 1726659151 ]] 2024-09-18T11:31:32.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-18T11:31:32.118 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:32.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:32.296 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:32.297 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-18T11:31:32.297 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:32.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:32.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:32.299 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:32.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659092 -lt 1726659151 ]] 2024-09-18T11:31:32.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-18T11:31:32.300 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:32.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:32.477 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:32.478 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:32.478 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-18T11:31:32.480 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:32.480 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:32.480 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:32.481 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659092 -lt 1726659151 ]] 2024-09-18T11:31:32.481 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-18T11:31:32.481 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:32.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:32.659 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:32.660 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-18T11:31:32.660 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:32.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:32.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:32.663 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:32.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659092 -lt 1726659151 ]] 2024-09-18T11:31:32.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-18T11:31:32.664 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:32.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:32.842 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:32.843 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:32.843 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-18T11:31:32.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:32.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:32.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:32.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659092 -lt 1726659151 ]] 2024-09-18T11:31:32.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-18T11:31:32.846 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:33.028 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:33.029 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:33.029 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:33.029 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-18T11:31:33.031 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:33.031 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:33.031 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:33.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659093 -lt 1726659151 ]] 2024-09-18T11:31:33.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-18T11:31:33.032 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:39.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:39.218 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:39.218 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:39.218 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-18T11:31:39.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:39.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:39.221 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:39.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659099 -lt 1726659151 ]] 2024-09-18T11:31:39.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-18T11:31:39.222 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:46.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:46.842 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:46.843 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:46.843 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-18T11:31:46.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-18T11:31:46.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-18T11:31:46.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-18T11:31:46.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-18T11:31:46.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-18T11:31:46.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-18T11:31:47.083 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-09-18T11:31:47.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-18T11:31:47.366 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (c,d,e) after stopping [b] 2024-09-18T11:31:47.375 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-18T11:31:50.554 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (b,d,e) after stopping [c] 2024-09-18T11:31:50.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-18T11:31:50.828 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (b,c,e) after stopping [d] 2024-09-18T11:31:50.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-18T11:31:51.063 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-09-18T11:31:51.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-18T11:31:51.331 INFO:tasks.workunit.client.0.smithi157.stderr:adding another mon that is not yet online will not break quorum 2024-09-18T11:31:51.340 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-18T11:31:51.573 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.a 2024-09-18T11:31:51.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-18T11:31:51.858 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.b 2024-09-18T11:31:51.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-18T11:31:52.119 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.c 2024-09-18T11:31:52.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-18T11:31:52.396 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.d 2024-09-18T11:31:52.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-18T11:31:55.584 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.e 2024-09-18T11:31:55.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2024-09-18T11:31:55.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:31:55.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:31:55.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:31:55.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:31:55.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:31:55.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:31:55.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2024-09-18T11:31:55.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-18T11:31:55.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-09-18T11:31:55.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-18T11:31:55.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-09-18T11:31:55.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-18T11:31:55.700 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-18T11:31:55.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726659175 2024-09-18T11:31:55.700 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:55.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659115 -lt 1726659175 ]] 2024-09-18T11:31:55.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:31:55.701 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:55.881 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:55.881 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:55.881 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:31:55.881 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:55.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:55.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:55.884 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:55.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659115 -lt 1726659175 ]] 2024-09-18T11:31:55.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:31:55.885 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:56.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:56.064 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:56.064 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:56.064 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:31:56.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:56.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:56.067 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:56.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659116 -lt 1726659175 ]] 2024-09-18T11:31:56.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:31:56.068 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:56.247 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:56.247 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:56.248 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:56.248 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:31:56.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:56.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:56.250 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:56.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659116 -lt 1726659175 ]] 2024-09-18T11:31:56.252 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:31:56.252 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:59.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:59.438 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:59.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:59.439 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:31:59.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:59.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:59.441 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:59.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659119 -lt 1726659175 ]] 2024-09-18T11:31:59.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:31:59.442 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:59.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:59.622 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:59.622 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:59.623 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:31:59.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:59.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:59.625 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:59.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659119 -lt 1726659175 ]] 2024-09-18T11:31:59.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:31:59.626 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:59.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:59.804 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:59.805 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:31:59.805 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:59.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:59.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:59.807 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:59.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659119 -lt 1726659175 ]] 2024-09-18T11:31:59.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:31:59.808 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:31:59.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:31:59.986 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:59.987 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:31:59.987 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:31:59.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:31:59.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:31:59.990 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:31:59.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659119 -lt 1726659175 ]] 2024-09-18T11:31:59.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:31:59.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:32:00.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:32:00.170 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:32:00.171 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:32:00.171 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:32:00.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:32:00.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:32:00.173 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:32:00.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659120 -lt 1726659175 ]] 2024-09-18T11:32:00.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:32:00.175 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:32:00.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:32:00.354 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:32:00.354 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:32:00.355 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:32:00.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:32:00.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:32:00.357 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:32:00.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659120 -lt 1726659175 ]] 2024-09-18T11:32:00.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:32:00.358 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:32:00.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:32:00.537 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:32:00.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:32:00.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:32:00.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:32:00.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:32:00.540 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:32:00.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659120 -lt 1726659175 ]] 2024-09-18T11:32:00.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:32:00.541 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:32:00.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:32:00.722 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:32:00.722 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:32:00.722 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:32:00.724 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:32:00.724 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:32:00.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:32:00.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659120 -lt 1726659175 ]] 2024-09-18T11:32:00.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:32:00.726 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:32:00.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:32:00.905 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:32:00.905 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:32:00.905 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:32:00.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:32:00.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:32:00.907 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:32:00.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659120 -lt 1726659175 ]] 2024-09-18T11:32:00.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:32:00.909 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:32:01.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:32:01.109 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:32:01.109 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:32:01.109 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:32:01.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:32:01.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:32:01.111 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:32:01.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659121 -lt 1726659175 ]] 2024-09-18T11:32:01.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:32:01.112 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:32:01.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:32:01.292 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:32:01.293 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:32:01.293 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:32:01.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-18T11:32:01.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-18T11:32:01.296 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-18T11:32:01.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726659121 -lt 1726659175 ]] 2024-09-18T11:32:01.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-18T11:32:01.297 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-18T11:32:08.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-18T11:32:08.187 INFO:tasks.workunit.client.0.smithi157.stderr:{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:32:08.187 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"5be5d46b-e374-428a-af2b-68c67ab24ab2","modified":"2024-09-18T11:31:18.682299Z","created":"2024-09-18T11:31:18.682299Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-18T11:32:08.187 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-18T11:32:08.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-18T11:32:08.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-18T11:32:08.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-18T11:32:08.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-18T11:32:08.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-18T11:32:08.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-18T11:32:11.369 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-09-18T11:32:11.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-18T11:32:11.563 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-09-18T11:32:11.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-18T11:32:14.704 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-09-18T11:32:14.706 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-18T11:32:17.847 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-09-18T11:32:17.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-18T11:32:18.108 INFO:tasks.workunit.client.0.smithi157.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-09-18T11:32:18.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-18T11:32:21.256 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-18T11:32:21.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-18T11:32:21.537 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.a 2024-09-18T11:32:21.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-18T11:32:27.684 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: removing mon.b would break quorum 2024-09-18T11:32:27.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-18T11:32:27.916 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: removing mon.c would break quorum 2024-09-18T11:32:27.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-18T11:32:31.057 INFO:tasks.workunit.client.0.smithi157.stderr:Error EBUSY: removing mon.d would break quorum 2024-09-18T11:32:31.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-18T11:32:31.366 INFO:tasks.workunit.client.0.smithi157.stderr:safe to remove mon.e 2024-09-18T11:32:31.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-18T11:32:31.376 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:32:31.377 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:32:31.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:32:31.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:32:31.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:32:31.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:32:31.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-18T11:32:31.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-18T11:32:31.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:32:31.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-18T11:32:31.485 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:32:31.485 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:32:31.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:32:31.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:32:31.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:32:31.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:32:31.490 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:32:31.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:32:31.491 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:32:31.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:32:31.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:32:31.492 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:32:31.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:32:31.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:32:31.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:32:31.493 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:32:31.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:32:31.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:32:31.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-18T11:32:31.507 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:32:31.507 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:32:31.507 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:32:31.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53871 2024-09-18T11:32:31.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:32:31.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:32:31.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T11:32:31.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/ok-to-stop 0 2024-09-18T11:32:31.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-18T11:32:31.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T11:32:31.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-18T11:32:31.508 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:32:31.508 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:32:31.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:32:31.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:32:31.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:32:31.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:32:31.508 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:32:31.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:32:31.509 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:32:31.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:32:31.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:32:31.510 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:32:31.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:32:31.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:32:31.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:32:31.511 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:32:31.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:32:31.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T11:32:31.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-18T11:32:31.513 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:32:31.513 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:32:31.513 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53871 2024-09-18T11:32:31.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53871 2024-09-18T11:32:31.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:32:31.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:32:31.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T11:32:31.515 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T11:32:31.516 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T11:32:31.581 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2024-09-18T11:32:31.582 DEBUG:teuthology.orchestra.run.smithi157:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2024-09-18T11:32:31.643 INFO:tasks.workunit.client.0.smithi157.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T11:32:31.649 INFO:tasks.workunit.client.0.smithi157.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T11:32:31.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T11:32:31.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T11:32:31.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T11:32:31.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T11:32:31.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T11:32:31.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T11:32:31.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T11:32:31.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T11:32:31.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/rados-striper 2024-09-18T11:32:31.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2024-09-18T11:32:31.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2024-09-18T11:32:31.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2024-09-18T11:32:31.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2024-09-18T11:32:31.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2024-09-18T11:32:31.651 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2024-09-18T11:32:31.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=7a5f0660-5673-4fa0-8301-af030aa45b03 --auth-supported=none ' 2024-09-18T11:32:31.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2024-09-18T11:32:31.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2024-09-18T11:32:31.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/rados-striper 2024-09-18T11:32:31.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/rados-striper 2024-09-18T11:32:31.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-09-18T11:32:31.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:32:31.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-09-18T11:32:31.652 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:32:31.652 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:32:31.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:32:31.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:32:31.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:32:31.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:32:31.654 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:32:31.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:32:31.654 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:32:31.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:32:31.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:32:31.654 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:32:31.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:32:31.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:32:31.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:32:31.655 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:32:31.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:32:31.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:32:31.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-09-18T11:32:31.656 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:32:31.656 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:32:31.656 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70837 2024-09-18T11:32:31.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70837 2024-09-18T11:32:31.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:32:31.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:32:31.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/rados-striper 2024-09-18T11:32:31.658 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:32:31.658 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:32:31.658 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70837 2024-09-18T11:32:31.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70837 2024-09-18T11:32:31.660 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:32:31.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T11:32:31.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T11:32:31.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:32:31.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2024-09-18T11:32:31.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2024-09-18T11:32:31.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/rados-striper 2024-09-18T11:32:31.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:32:31.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:32:31.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:32:31.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/rados-striper/a 2024-09-18T11:32:31.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2024-09-18T11:32:32.016 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:32:32.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:32:32.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:32:32.017 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:32:32.017 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:32:32.017 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70837 2024-09-18T11:32:32.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70837/$cluster-$name.asok' 2024-09-18T11:32:32.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.70837/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:32:32.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:32:32.277 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:32:32.277 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:32:32.277 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:32:32.277 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:32:32.277 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:32:32.278 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:32:32.278 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:32:32.278 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:32:32.278 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:32:32.278 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:32:32.278 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70837 2024-09-18T11:32:32.278 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70837/ceph-mon.a.asok 2024-09-18T11:32:32.278 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:32:32.278 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70837/ceph-mon.a.asok config get fsid 2024-09-18T11:32:32.351 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:32:32.351 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:32:32.352 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:32:32.352 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:32:32.352 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:32:32.352 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:32:32.352 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:32:32.352 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:32:32.352 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:32:32.352 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:32:32.352 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70837 2024-09-18T11:32:32.352 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70837/ceph-mon.a.asok 2024-09-18T11:32:32.352 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:32:32.352 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70837/ceph-mon.a.asok config get mon_host 2024-09-18T11:32:32.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2024-09-18T11:32:32.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-09-18T11:32:32.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:32:32.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:32:32.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:32:32.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/0 2024-09-18T11:32:32.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7a5f0660-5673-4fa0-8301-af030aa45b03 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-18T11:32:32.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:32:32.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:32:32.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:32:32.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2024-09-18T11:32:32.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2024-09-18T11:32:32.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:32:32.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:32:32.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-18T11:32:32.429 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:32:32.429 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:32:32.429 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:32:32.429 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:32:32.429 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:32:32.429 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70837 2024-09-18T11:32:32.429 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70837/$cluster-$name.asok' 2024-09-18T11:32:32.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70837/$cluster-$name.asok' 2024-09-18T11:32:32.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:32:32.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:32:32.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:32:32.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-18T11:32:32.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-18T11:32:32.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:32:32.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:32:32.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:32:32.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:32:32.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:32:32.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:32:32.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/0 2024-09-18T11:32:32.430 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:32:32.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8c686b55-9f26-4604-92f8-b7be15e340f4 2024-09-18T11:32:32.431 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 8c686b55-9f26-4604-92f8-b7be15e340f4 2024-09-18T11:32:32.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8c686b55-9f26-4604-92f8-b7be15e340f4' 2024-09-18T11:32:32.431 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:32:32.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBQuupmU7c6GhAAf23Abd2cUcGDlndCXKxCCQ== 2024-09-18T11:32:32.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBQuupmU7c6GhAAf23Abd2cUcGDlndCXKxCCQ=="}' 2024-09-18T11:32:32.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8c686b55-9f26-4604-92f8-b7be15e340f4 -i td/rados-striper/0/new.json 2024-09-18T11:32:33.305 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:32:33.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/0/new.json 2024-09-18T11:32:33.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7a5f0660-5673-4fa0-8301-af030aa45b03 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70837/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQuupmU7c6GhAAf23Abd2cUcGDlndCXKxCCQ== --osd-uuid 8c686b55-9f26-4604-92f8-b7be15e340f4 2024-09-18T11:32:33.337 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:32:33.335+0000 7fad8f12b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:32:33.352 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:32:33.350+0000 7fad8f12b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:32:33.352 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:32:33.351+0000 7fad8f12b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:32:33.353 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:32:33.352+0000 7fad8f12b440 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-09-18T11:32:33.353 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:32:33.352+0000 7fad8f12b440 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-09-18T11:32:33.353 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:32:33.352+0000 7fad8f12b440 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-09-18T11:32:33.392 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:32:33.391+0000 7fad8f12b440 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2024-09-18T11:32:38.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/0/keyring 2024-09-18T11:32:38.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:32:38.712 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:32:38.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:32:38.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:32:38.905 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:32:38.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:32:38.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7a5f0660-5673-4fa0-8301-af030aa45b03 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70837/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:32:38.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:32:38.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:32:38.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:32:38.926 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:32:38.924+0000 7fe8ac123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:32:38.966 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:32:38.965+0000 7fe8ac123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:32:38.967 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:32:38.966+0000 7fe8ac123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:32:39.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:32:39.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:32:39.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:32:39.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:32:39.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:32:39.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:39.080 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:32:39.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:32:39.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:39.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:39.289 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:39.497 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:32:39.496+0000 7fe8ac123440 -1 Falling back to public interface 2024-09-18T11:32:40.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:40.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:40.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:32:40.291 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:32:40.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:40.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:40.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:41.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:41.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:41.469 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:32:41.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:32:41.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:41.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:41.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:42.111 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:32:42.108+0000 7fe8ac123440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:32:42.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:42.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:42.644 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:32:42.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:32:42.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:42.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:42.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:43.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:43.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:43.888 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:32:43.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:32:43.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:43.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:44.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:45.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:45.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:45.071 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:32:45.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:32:45.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:45.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:45.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:46.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:46.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:46.257 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:32:46.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:32:46.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:46.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:46.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:47.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:47.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:47.442 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:32:47.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:32:47.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:47.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:47.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:48.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:48.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:48.629 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:32:48.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:32:48.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:48.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:48.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:49.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:49.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:49.813 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:32:49.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:32:49.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:49.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:49.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:50.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:50.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:50.996 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:32:50.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:32:50.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:50.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:51.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:52.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:52.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:52.184 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:32:52.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:32:52.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:52.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:52.368 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:53.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:53.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:53.371 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:32:53.371 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:32:53.371 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:53.371 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:53.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:54.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:54.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:54.559 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:32:54.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:32:54.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:54.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:54.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:55.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:55.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:55.748 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:32:55.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:32:55.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:55.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:55.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:56.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:56.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:56.934 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:32:56.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:32:56.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:56.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:57.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:58.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:58.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:58.116 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:32:58.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:32:58.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:58.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:58.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:59.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:59.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:59.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:32:59.301 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:32:59.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:59.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:59.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:00.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:00.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:00.484 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T11:33:00.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:33:00.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:00.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:33:00.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:01.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:01.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:01.665 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-18T11:33:01.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T11:33:01.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:01.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:33:01.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:02.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:02.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:02.852 INFO:tasks.workunit.client.0.smithi157.stdout:20 2024-09-18T11:33:02.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T11:33:02.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:02.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:33:03.165 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/71780,v1:127.0.0.1:6801/71780] [v2:127.0.0.1:6802/71780,v1:127.0.0.1:6803/71780] exists,up 8c686b55-9f26-4604-92f8-b7be15e340f4 2024-09-18T11:33:03.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:33:03.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:33:03.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:33:03.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2024-09-18T11:33:03.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-09-18T11:33:03.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:33:03.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:33:03.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:33:03.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/1 2024-09-18T11:33:03.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7a5f0660-5673-4fa0-8301-af030aa45b03 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-18T11:33:03.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:33:03.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:33:03.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:33:03.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2024-09-18T11:33:03.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2024-09-18T11:33:03.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:33:03.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:33:03.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-18T11:33:03.171 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:33:03.171 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:33:03.171 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:33:03.171 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:33:03.171 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:33:03.171 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70837 2024-09-18T11:33:03.171 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70837/$cluster-$name.asok' 2024-09-18T11:33:03.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70837/$cluster-$name.asok' 2024-09-18T11:33:03.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:33:03.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:33:03.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:33:03.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-18T11:33:03.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-18T11:33:03.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:33:03.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:33:03.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:33:03.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:33:03.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:33:03.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:33:03.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/1 2024-09-18T11:33:03.172 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:33:03.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=74f1d5eb-1f23-462b-b30e-3bfc5c645cc7 2024-09-18T11:33:03.173 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 74f1d5eb-1f23-462b-b30e-3bfc5c645cc7 2024-09-18T11:33:03.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 74f1d5eb-1f23-462b-b30e-3bfc5c645cc7' 2024-09-18T11:33:03.173 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:33:03.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvuupmXmvLChAAZYtyrQ5qf9y+YD2UJDot0A== 2024-09-18T11:33:03.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvuupmXmvLChAAZYtyrQ5qf9y+YD2UJDot0A=="}' 2024-09-18T11:33:03.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 74f1d5eb-1f23-462b-b30e-3bfc5c645cc7 -i td/rados-striper/1/new.json 2024-09-18T11:33:03.516 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:33:03.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/1/new.json 2024-09-18T11:33:03.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=7a5f0660-5673-4fa0-8301-af030aa45b03 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70837/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvuupmXmvLChAAZYtyrQ5qf9y+YD2UJDot0A== --osd-uuid 74f1d5eb-1f23-462b-b30e-3bfc5c645cc7 2024-09-18T11:33:03.548 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:03.547+0000 7f6a2e20d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:03.550 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:03.548+0000 7f6a2e20d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:03.551 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:03.549+0000 7f6a2e20d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:03.551 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:03.550+0000 7f6a2e20d440 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-09-18T11:33:03.551 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:03.550+0000 7f6a2e20d440 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-09-18T11:33:03.551 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:03.550+0000 7f6a2e20d440 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-09-18T11:33:03.625 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:03.624+0000 7f6a2e20d440 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2024-09-18T11:33:09.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/1/keyring 2024-09-18T11:33:09.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:33:09.467 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T11:33:09.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:33:09.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:33:09.659 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T11:33:09.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:33:09.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=7a5f0660-5673-4fa0-8301-af030aa45b03 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70837/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:33:09.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:33:09.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:33:09.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:33:09.680 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:09.678+0000 7fa91171e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:09.686 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:09.685+0000 7fa91171e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:09.687 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:09.686+0000 7fa91171e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:09.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:33:09.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:33:09.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:33:09.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:33:09.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:33:09.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:09.860 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:33:09.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:33:09.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:09.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:10.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:10.219 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:10.218+0000 7fa91171e440 -1 Falling back to public interface 2024-09-18T11:33:11.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:11.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:11.058 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:33:11.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:33:11.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:11.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:11.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:12.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:12.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:12.239 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:33:12.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:33:12.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:12.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:12.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:12.821 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:12.820+0000 7fa91171e440 -1 osd.1 0 log_to_monitors true 2024-09-18T11:33:13.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:13.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:13.426 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:33:13.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:33:13.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:13.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:13.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:14.648 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:14.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:14.649 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:33:14.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:33:14.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:14.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:14.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:15.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:15.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:15.836 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:33:15.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:33:15.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:15.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:16.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:17.019 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:17.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:17.020 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:33:17.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:33:17.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:17.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:17.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:18.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:18.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:18.204 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:33:18.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:33:18.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:18.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:18.390 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:19.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:19.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:19.392 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:33:19.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:33:19.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:19.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:19.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:20.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:20.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:20.578 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:33:20.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:33:20.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:20.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:20.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:21.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:21.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:21.763 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:33:21.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:33:21.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:21.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:21.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:22.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:22.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:22.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:33:22.948 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:33:22.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:22.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:23.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:24.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:24.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:24.133 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:33:24.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:33:24.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:24.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:24.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:25.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:25.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:25.318 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:33:25.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:33:25.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:25.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:25.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:26.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:26.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:26.505 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:33:26.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:33:26.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:26.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:26.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:27.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:27.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:27.693 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:33:27.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:33:27.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:27.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:27.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:28.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:28.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:28.878 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:33:28.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:33:28.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:28.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:29.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:30.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:30.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:30.064 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:33:30.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:33:30.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:30.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:30.247 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:31.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:31.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:31.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:33:31.249 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T11:33:31.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:31.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:31.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:32.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:32.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:32.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T11:33:32.434 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-18T11:33:32.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:32.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:32.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:33.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:33.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:33.619 INFO:tasks.workunit.client.0.smithi157.stdout:20 2024-09-18T11:33:33.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T11:33:33.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:33.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:33:33.844 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/73609,v1:127.0.0.1:6809/73609] [v2:127.0.0.1:6810/73609,v1:127.0.0.1:6811/73609] exists,up 74f1d5eb-1f23-462b-b30e-3bfc5c645cc7 2024-09-18T11:33:33.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:33:33.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:33:33.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:33:33.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2024-09-18T11:33:33.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-09-18T11:33:33.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:33:33.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:33:33.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:33:33.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/2 2024-09-18T11:33:33.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7a5f0660-5673-4fa0-8301-af030aa45b03 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-18T11:33:33.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:33:33.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:33:33.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:33:33.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2024-09-18T11:33:33.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2024-09-18T11:33:33.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:33:33.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:33:33.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-18T11:33:33.850 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:33:33.850 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:33:33.850 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:33:33.850 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:33:33.850 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:33:33.850 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70837 2024-09-18T11:33:33.850 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70837/$cluster-$name.asok' 2024-09-18T11:33:33.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70837/$cluster-$name.asok' 2024-09-18T11:33:33.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:33:33.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:33:33.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:33:33.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-18T11:33:33.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-18T11:33:33.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:33:33.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:33:33.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:33:33.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:33:33.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:33:33.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:33:33.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/2 2024-09-18T11:33:33.851 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:33:33.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=436a0399-5295-40e1-ae8a-6acfc2eff006 2024-09-18T11:33:33.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 436a0399-5295-40e1-ae8a-6acfc2eff006' 2024-09-18T11:33:33.852 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 436a0399-5295-40e1-ae8a-6acfc2eff006 2024-09-18T11:33:33.852 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:33:33.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCNuupmbxNJMxAAHKm+a9EoFFU/LWMuli4QFA== 2024-09-18T11:33:33.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCNuupmbxNJMxAAHKm+a9EoFFU/LWMuli4QFA=="}' 2024-09-18T11:33:33.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 436a0399-5295-40e1-ae8a-6acfc2eff006 -i td/rados-striper/2/new.json 2024-09-18T11:33:34.254 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:33:34.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/2/new.json 2024-09-18T11:33:34.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=7a5f0660-5673-4fa0-8301-af030aa45b03 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70837/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCNuupmbxNJMxAAHKm+a9EoFFU/LWMuli4QFA== --osd-uuid 436a0399-5295-40e1-ae8a-6acfc2eff006 2024-09-18T11:33:34.286 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:34.284+0000 7fb48cd62440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:34.288 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:34.286+0000 7fb48cd62440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:34.289 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:34.287+0000 7fb48cd62440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:34.289 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:34.288+0000 7fb48cd62440 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-09-18T11:33:34.289 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:34.288+0000 7fb48cd62440 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-09-18T11:33:34.289 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:34.288+0000 7fb48cd62440 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-09-18T11:33:34.362 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:34.361+0000 7fb48cd62440 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2024-09-18T11:33:39.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/2/keyring 2024-09-18T11:33:39.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:33:39.957 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-18T11:33:39.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:33:39.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:33:40.148 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-18T11:33:40.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:33:40.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=7a5f0660-5673-4fa0-8301-af030aa45b03 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70837/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:33:40.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:33:40.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:33:40.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:33:40.169 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:40.167+0000 7fa40570f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:40.172 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:40.171+0000 7fa40570f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:40.173 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:40.172+0000 7fa40570f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:40.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:33:40.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:33:40.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:33:40.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:33:40.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:33:40.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:40.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:33:40.345 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:33:40.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:40.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:40.553 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:40.701 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:40.700+0000 7fa40570f440 -1 Falling back to public interface 2024-09-18T11:33:41.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:41.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:41.555 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:33:41.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:33:41.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:41.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:41.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:42.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:42.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:42.736 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:33:42.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:33:42.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:42.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:42.920 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:43.282 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:33:43.281+0000 7fa40570f440 -1 osd.2 0 log_to_monitors true 2024-09-18T11:33:43.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:43.922 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:33:43.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:43.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:33:43.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:43.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:44.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:45.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:45.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:45.247 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:33:45.247 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:33:45.247 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:45.247 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:45.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:46.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:46.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:46.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:33:46.433 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:33:46.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:46.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:46.615 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:47.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:47.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:47.617 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:33:47.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:33:47.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:47.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:47.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:48.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:48.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:48.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:33:48.801 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:33:48.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:48.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:48.985 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:49.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:49.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:49.986 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:33:49.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:33:49.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:49.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:50.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:51.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:51.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:51.172 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:33:51.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:33:51.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:51.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:51.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:52.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:52.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:52.358 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:33:52.359 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:33:52.359 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:52.359 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:52.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:53.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:53.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:53.545 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:33:53.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:33:53.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:53.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:53.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:54.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:54.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:54.730 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:33:54.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:33:54.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:54.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:54.915 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:55.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:55.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:55.918 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:33:55.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:33:55.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:55.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:56.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:57.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:57.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:57.107 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:33:57.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:33:57.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:57.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:57.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:58.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:58.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:58.291 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:33:58.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:33:58.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:58.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:58.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:59.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:59.475 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:33:59.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:59.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:33:59.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:59.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:59.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:00.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:00.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:00.662 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:34:00.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:34:00.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:00.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:34:00.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:01.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:01.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:01.850 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T11:34:01.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:34:01.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:01.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:34:02.033 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:03.034 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:03.035 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:03.035 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-18T11:34:03.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T11:34:03.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:03.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:34:03.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:04.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:04.218 INFO:tasks.workunit.client.0.smithi157.stdout:20 2024-09-18T11:34:04.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:04.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T11:34:04.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:04.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:34:04.441 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/75438,v1:127.0.0.1:6817/75438] [v2:127.0.0.1:6818/75438,v1:127.0.0.1:6819/75438] exists,up 436a0399-5295-40e1-ae8a-6acfc2eff006 2024-09-18T11:34:04.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:34:04.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:34:04.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:34:04.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2024-09-18T11:34:04.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:34:04.701 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:34:04.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:34:04.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:34:05.019 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:34:05.028 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:34:06.030 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:34:08.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2024-09-18T11:34:08.466 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records in 2024-09-18T11:34:08.466 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records out 2024-09-18T11:34:08.466 INFO:tasks.workunit.client.0.smithi157.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 5.4762e-05 s, 22.5 MB/s 2024-09-18T11:34:08.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2024-09-18T11:34:09.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2024-09-18T11:34:09.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2024-09-18T11:34:09.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2024-09-18T11:34:09.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2024-09-18T11:34:09.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2024-09-18T11:34:09.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2024-09-18T11:34:09.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2024-09-18T11:34:09.280 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2024-09-18T11:34:09.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2024-09-18T11:34:09.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2024-09-18T11:34:09.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2024-09-18T11:34:09.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2024-09-18T11:34:09.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2024-09-18T11:34:09.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2024-09-18T11:34:10.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2024-09-18T11:34:10.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2024-09-18T11:34:10.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2024-09-18T11:34:10.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2024-09-18T11:34:10.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2024-09-18T11:34:10.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2024-09-18T11:34:10.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2024-09-18T11:34:10.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2024-09-18T11:34:10.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2024-09-18T11:34:10.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-18T11:34:10.293 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2024-09-18T11:34:10.293 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2024-09-18T11:34:10.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2024-09-18T11:34:10.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2024-09-18T11:34:10.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2024-09-18T11:34:10.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2024-09-18T11:34:10.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2024-09-18T11:34:10.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2024-09-18T11:34:10.340 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2024-09-18T11:34:10.340 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2024-09-18T11:34:10.480 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2024-09-18T11:34:10.480 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2024-09-18T11:34:10.481 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2024-09-18T11:34:10.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2024-09-18T11:34:10.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/rados-striper 2024-09-18T11:34:10.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-18T11:34:10.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local 'expected=No data available' 2024-09-18T11:34:10.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-18T11:34:10.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-18T11:34:10.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2024-09-18T11:34:10.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-18T11:34:10.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-18T11:34:10.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-09-18T11:34:10.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: expect_failure: return 0 2024-09-18T11:34:10.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-18T11:34:10.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/rados-striper 2024-09-18T11:34:10.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-18T11:34:10.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local 'expected=No data available' 2024-09-18T11:34:10.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-18T11:34:10.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-18T11:34:10.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-18T11:34:10.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-18T11:34:10.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-18T11:34:10.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-09-18T11:34:10.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: expect_failure: return 0 2024-09-18T11:34:10.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2024-09-18T11:34:10.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2024-09-18T11:34:10.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/rados-striper 2024-09-18T11:34:10.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-18T11:34:10.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local 'expected=No such file or directory' 2024-09-18T11:34:10.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-18T11:34:10.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-18T11:34:10.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: rados --pool rbd --striper stat toyfile 2024-09-18T11:34:10.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-18T11:34:10.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-18T11:34:10.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-09-18T11:34:10.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: expect_failure: return 0 2024-09-18T11:34:10.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2024-09-18T11:34:10.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/rados-striper 2024-09-18T11:34:10.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-18T11:34:10.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local 'expected=No such file or directory' 2024-09-18T11:34:10.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-18T11:34:10.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-18T11:34:10.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2024-09-18T11:34:10.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-18T11:34:10.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-18T11:34:10.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-09-18T11:34:10.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: expect_failure: return 0 2024-09-18T11:34:10.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2024-09-18T11:34:10.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-09-18T11:34:10.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:34:10.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-09-18T11:34:10.779 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:34:10.779 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:34:10.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:34:10.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:34:10.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:34:10.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:34:10.885 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:34:10.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:34:10.886 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:34:10.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:34:10.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:34:10.888 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:34:10.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:34:10.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:34:10.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:34:10.889 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:34:10.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:34:10.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:34:10.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-09-18T11:34:10.902 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:34:10.902 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:10.902 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70837 2024-09-18T11:34:10.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70837 2024-09-18T11:34:10.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:34:10.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:34:10.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T11:34:10.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/rados-striper 0 2024-09-18T11:34:10.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-09-18T11:34:10.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T11:34:10.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-09-18T11:34:10.904 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:34:10.904 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:34:10.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:34:10.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:34:10.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:34:10.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:34:10.905 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:34:10.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:34:10.906 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:34:10.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:34:10.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:34:10.907 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:34:10.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:34:10.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:34:10.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:34:10.909 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:34:10.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:34:10.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T11:34:10.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-09-18T11:34:10.911 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:34:10.911 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:10.912 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70837 2024-09-18T11:34:10.912 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70837 2024-09-18T11:34:10.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:34:10.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:34:10.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T11:34:10.914 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T11:34:10.914 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T11:34:10.979 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2024-09-18T11:34:10.979 DEBUG:teuthology.orchestra.run.smithi157:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2024-09-18T11:34:11.042 INFO:tasks.workunit.client.0.smithi157.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T11:34:11.047 INFO:tasks.workunit.client.0.smithi157.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T11:34:11.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: run_tests: export .:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T11:34:11.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2299: export: `.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin': not a valid identifier 2024-09-18T11:34:11.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2301: run_tests: export CEPH_MON=127.0.0.1:7109 2024-09-18T11:34:11.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2301: run_tests: CEPH_MON=127.0.0.1:7109 2024-09-18T11:34:11.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2302: run_tests: export CEPH_ARGS 2024-09-18T11:34:11.047 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2303: run_tests: uuidgen 2024-09-18T11:34:11.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2303: run_tests: CEPH_ARGS+=' --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none ' 2024-09-18T11:34:11.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2304: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2024-09-18T11:34:11.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2305: run_tests: export CEPH_CONF=/dev/null 2024-09-18T11:34:11.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2305: run_tests: CEPH_CONF=/dev/null 2024-09-18T11:34:11.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: run_tests: set 2024-09-18T11:34:11.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2024-09-18T11:34:11.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: run_tests: local 'funcs=test_activate_osd 2024-09-18T11:34:11.051 INFO:tasks.workunit.client.0.smithi157.stderr:test_activate_osd_after_mark_down 2024-09-18T11:34:11.051 INFO:tasks.workunit.client.0.smithi157.stderr:test_activate_osd_skip_benchmark 2024-09-18T11:34:11.051 INFO:tasks.workunit.client.0.smithi157.stderr:test_destroy_osd 2024-09-18T11:34:11.051 INFO:tasks.workunit.client.0.smithi157.stderr:test_display_logs 2024-09-18T11:34:11.051 INFO:tasks.workunit.client.0.smithi157.stderr:test_erasure_code_plugin_exists 2024-09-18T11:34:11.051 INFO:tasks.workunit.client.0.smithi157.stderr:test_expect_failure 2024-09-18T11:34:11.051 INFO:tasks.workunit.client.0.smithi157.stderr:test_flush_pg_stats 2024-09-18T11:34:11.051 INFO:tasks.workunit.client.0.smithi157.stderr:test_get_config 2024-09-18T11:34:11.051 INFO:tasks.workunit.client.0.smithi157.stderr:test_get_is_making_recovery_progress 2024-09-18T11:34:11.051 INFO:tasks.workunit.client.0.smithi157.stderr:test_get_last_scrub_stamp 2024-09-18T11:34:11.051 INFO:tasks.workunit.client.0.smithi157.stderr:test_get_not_primary 2024-09-18T11:34:11.051 INFO:tasks.workunit.client.0.smithi157.stderr:test_get_num_active_clean 2024-09-18T11:34:11.051 INFO:tasks.workunit.client.0.smithi157.stderr:test_get_num_active_or_peered 2024-09-18T11:34:11.052 INFO:tasks.workunit.client.0.smithi157.stderr:test_get_num_pgs 2024-09-18T11:34:11.052 INFO:tasks.workunit.client.0.smithi157.stderr:test_get_op_scheduler 2024-09-18T11:34:11.052 INFO:tasks.workunit.client.0.smithi157.stderr:test_get_osd_id_used_by_pgs 2024-09-18T11:34:11.052 INFO:tasks.workunit.client.0.smithi157.stderr:test_get_osds 2024-09-18T11:34:11.052 INFO:tasks.workunit.client.0.smithi157.stderr:test_get_pg 2024-09-18T11:34:11.052 INFO:tasks.workunit.client.0.smithi157.stderr:test_get_primary 2024-09-18T11:34:11.052 INFO:tasks.workunit.client.0.smithi157.stderr:test_get_timeout_delays 2024-09-18T11:34:11.052 INFO:tasks.workunit.client.0.smithi157.stderr:test_is_clean 2024-09-18T11:34:11.052 INFO:tasks.workunit.client.0.smithi157.stderr:test_kill_daemon 2024-09-18T11:34:11.052 INFO:tasks.workunit.client.0.smithi157.stderr:test_kill_daemons 2024-09-18T11:34:11.053 INFO:tasks.workunit.client.0.smithi157.stderr:test_objectstore_tool 2024-09-18T11:34:11.053 INFO:tasks.workunit.client.0.smithi157.stderr:test_pg_scrub 2024-09-18T11:34:11.053 INFO:tasks.workunit.client.0.smithi157.stderr:test_repair 2024-09-18T11:34:11.053 INFO:tasks.workunit.client.0.smithi157.stderr:test_run_in_background 2024-09-18T11:34:11.053 INFO:tasks.workunit.client.0.smithi157.stderr:test_run_mon 2024-09-18T11:34:11.053 INFO:tasks.workunit.client.0.smithi157.stderr:test_run_osd 2024-09-18T11:34:11.053 INFO:tasks.workunit.client.0.smithi157.stderr:test_set_config 2024-09-18T11:34:11.053 INFO:tasks.workunit.client.0.smithi157.stderr:test_setup 2024-09-18T11:34:11.053 INFO:tasks.workunit.client.0.smithi157.stderr:test_teardown 2024-09-18T11:34:11.053 INFO:tasks.workunit.client.0.smithi157.stderr:test_wait_background 2024-09-18T11:34:11.053 INFO:tasks.workunit.client.0.smithi157.stderr:test_wait_for_clean 2024-09-18T11:34:11.053 INFO:tasks.workunit.client.0.smithi157.stderr:test_wait_for_health_ok 2024-09-18T11:34:11.053 INFO:tasks.workunit.client.0.smithi157.stderr:test_wait_for_osd 2024-09-18T11:34:11.054 INFO:tasks.workunit.client.0.smithi157.stderr:test_wait_for_peered 2024-09-18T11:34:11.054 INFO:tasks.workunit.client.0.smithi157.stderr:test_wait_for_scrub 2024-09-18T11:34:11.054 INFO:tasks.workunit.client.0.smithi157.stderr:test_wait_osd_id_used_by_pgs' 2024-09-18T11:34:11.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: run_tests: local dir=td/ceph-helpers 2024-09-18T11:34:11.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:34:11.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_activate_osd td/ceph-helpers 2024-09-18T11:34:11.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local dir=td/ceph-helpers 2024-09-18T11:34:11.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:888: test_activate_osd: setup td/ceph-helpers 2024-09-18T11:34:11.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:34:11.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:34:11.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:34:11.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:34:11.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:34:11.055 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:34:11.055 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:34:11.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:34:11.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:34:11.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:34:11.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:34:11.056 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:34:11.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:34:11.056 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:34:11.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:34:11.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:34:11.057 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:34:11.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:34:11.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:34:11.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:34:11.057 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:34:11.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:34:11.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:34:11.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:34:11.058 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:34:11.058 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:11.058 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:11.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:34:11.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:34:11.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:34:11.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:34:11.060 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:34:11.060 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:11.060 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:11.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:34:11.062 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:34:11.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T11:34:11.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T11:34:11.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:34:11.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:34:11.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:890: test_activate_osd: run_mon td/ceph-helpers a 2024-09-18T11:34:11.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:34:11.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:34:11.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:34:11.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:34:11.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:34:11.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-18T11:34:11.429 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:34:11.429 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:34:11.429 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:34:11.429 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:34:11.429 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:11.429 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:11.429 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:34:11.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:34:11.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:34:11.656 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:34:11.656 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:34:11.657 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:34:11.657 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:34:11.658 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:34:11.658 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:34:11.658 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:34:11.658 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:34:11.658 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:34:11.658 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:11.658 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:11.658 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:34:11.658 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:34:11.658 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:34:11.742 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:34:11.743 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:34:11.743 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:34:11.743 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:34:11.743 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:34:11.743 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:34:11.743 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:34:11.743 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:34:11.743 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:34:11.743 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:11.743 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:11.743 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:34:11.743 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:34:11.744 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:34:11.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: run_mgr td/ceph-helpers x 2024-09-18T11:34:11.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:34:11.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:34:11.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:34:11.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:34:11.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:34:11.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:34:12.667 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:34:12.667 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:34:12.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:34:12.668 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:34:12.668 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:12.668 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:12.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:34:12.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:34:12.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:34:12.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: run_osd td/ceph-helpers 0 2024-09-18T11:34:12.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:34:12.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:34:12.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:34:12.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:34:12.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:34:12.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:34:12.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:34:12.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:34:12.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:34:12.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:34:12.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:34:12.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:34:12.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:34:12.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:34:12.696 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:34:12.696 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:34:12.696 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:34:12.696 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:34:12.696 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:12.696 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:12.697 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:34:12.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:34:12.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:34:12.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:34:12.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:34:12.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:34:12.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:34:12.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:34:12.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:34:12.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:34:12.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:34:12.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:34:12.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:34:12.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:34:12.698 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:34:12.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=98e933fa-6f2a-4fc9-9469-64c6121a6813 2024-09-18T11:34:12.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 98e933fa-6f2a-4fc9-9469-64c6121a6813' 2024-09-18T11:34:12.698 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 98e933fa-6f2a-4fc9-9469-64c6121a6813 2024-09-18T11:34:12.698 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:34:12.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0uupmO2c+KhAA7PplxmvpEbI5uXcIgmnteQ== 2024-09-18T11:34:12.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0uupmO2c+KhAA7PplxmvpEbI5uXcIgmnteQ=="}' 2024-09-18T11:34:12.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 98e933fa-6f2a-4fc9-9469-64c6121a6813 -i td/ceph-helpers/0/new.json 2024-09-18T11:34:13.010 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:34:13.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:34:13.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0uupmO2c+KhAA7PplxmvpEbI5uXcIgmnteQ== --osd-uuid 98e933fa-6f2a-4fc9-9469-64c6121a6813 2024-09-18T11:34:13.045 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:13.043+0000 7f51b2412440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:13.056 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:13.055+0000 7f51b2412440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:13.057 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:13.056+0000 7f51b2412440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:13.057 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:13.056+0000 7f51b2412440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:34:13.057 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:13.056+0000 7f51b2412440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:34:13.058 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:13.056+0000 7f51b2412440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:34:13.096 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:13.095+0000 7f51b2412440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:34:18.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:34:18.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:34:18.438 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:34:18.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:34:18.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:34:18.923 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:34:18.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:34:18.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:34:18.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:34:18.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:34:18.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:34:18.945 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:18.944+0000 7fc776e87440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:18.947 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:18.946+0000 7fc776e87440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:18.948 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:18.947+0000 7fc776e87440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:19.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:34:19.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:34:19.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:34:19.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:34:19.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:34:19.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:19.279 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:34:19.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:34:19.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:19.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:19.470 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:19.469+0000 7fc776e87440 -1 Falling back to public interface 2024-09-18T11:34:19.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:20.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:20.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:20.637 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:34:20.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:34:20.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:20.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:20.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:21.987 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:21.986+0000 7fc776e87440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:34:21.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:21.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:21.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:34:21.992 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:34:21.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:21.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:22.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:23.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:23.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:23.357 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:34:23.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:34:23.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:23.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:23.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:24.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:24.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:24.808 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:34:24.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:34:24.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:24.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:25.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:26.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:26.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:26.192 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:34:26.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:34:26.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:26.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:26.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:27.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:27.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:27.687 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:34:27.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:34:27.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:27.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:28.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:29.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:29.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:29.081 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:34:29.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:34:29.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:29.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:29.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:30.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:30.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:30.441 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:34:30.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:34:30.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:30.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:30.988 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:31.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:31.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:31.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:34:31.990 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:34:31.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:31.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:32.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:33.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:33.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:33.387 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:34:33.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:34:33.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:33.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:34.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:35.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:35.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:35.433 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:34:35.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:34:35.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:35.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:35.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:36.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:36.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:36.974 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:34:36.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:34:36.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:36.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:37.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:38.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:38.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:38.680 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:34:38.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:34:38.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:38.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:39.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:40.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:40.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:40.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:34:40.420 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:34:40.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:40.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:41.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:42.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:42.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:42.149 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:34:42.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:34:42.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:42.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:42.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:43.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:43.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:43.628 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:34:43.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:34:43.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:43.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:44.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:45.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:45.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:45.145 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:34:45.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:34:45.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:45.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:45.504 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/78086,v1:127.0.0.1:6803/78086] [v2:127.0.0.1:6804/78086,v1:127.0.0.1:6805/78086] exists,up 98e933fa-6f2a-4fc9-9469-64c6121a6813 2024-09-18T11:34:45.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:34:45.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:34:45.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:34:45.506 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: get_asok_path osd.0 2024-09-18T11:34:45.506 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:34:45.506 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:34:45.506 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:34:45.506 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:45.506 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:45.506 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-osd.0.asok 2024-09-18T11:34:45.506 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: CEPH_ARGS= 2024-09-18T11:34:45.506 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.76995/ceph-osd.0.asok config get osd_max_backfills 2024-09-18T11:34:45.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-09-18T11:34:45.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2024-09-18T11:34:45.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: grep --quiet osd_max_backfills 2024-09-18T11:34:45.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:898: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2024-09-18T11:34:45.587 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:34:45.587 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:34:45.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:34:45.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:34:45.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:34:45.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:34:45.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-09-18T11:34:45.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-18T11:34:45.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:34:45.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T11:34:45.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:34:45.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:34:45.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:34:45.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:34:45.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:34:45.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:34:45.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:34:45.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:34:45.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:34:45.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:34:45.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:34:45.900 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:34:45.900 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:34:45.900 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:34:45.900 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:34:45.900 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:45.900 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:45.900 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:34:45.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:34:45.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:34:45.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:34:45.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:34:45.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:34:45.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:34:45.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:34:45.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:34:45.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:34:45.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-09-18T11:34:45.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:34:45.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T11:34:45.902 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:34:45.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-09-18T11:34:45.903 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-18T11:34:45.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T11:34:45.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:34:45.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:34:45.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:34:45.918 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:45.917+0000 7f9f98009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:45.920 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:45.919+0000 7f9f98009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:45.921 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:45.920+0000 7f9f98009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:46.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T11:34:46.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:34:46.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:34:46.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:34:46.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:34:46.298 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:34:46.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:46.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:34:46.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:46.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:46.444 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:46.442+0000 7f9f98009440 -1 Falling back to public interface 2024-09-18T11:34:46.648 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:47.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:47.649 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:34:47.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:47.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:34:47.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:47.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:48.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:49.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:49.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:49.059 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:34:49.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:34:49.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:49.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:49.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:49.485 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:49.483+0000 7f9f98009440 -1 osd.0 6 log_to_monitors true 2024-09-18T11:34:50.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:50.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:50.421 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:34:50.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:34:50.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:50.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:50.772 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/79029,v1:127.0.0.1:6803/79029] [v2:127.0.0.1:6804/79029,v1:127.0.0.1:6805/79029] exists,up 98e933fa-6f2a-4fc9-9469-64c6121a6813 2024-09-18T11:34:50.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:34:50.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:34:50.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:34:50.774 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: get_asok_path osd.0 2024-09-18T11:34:50.774 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:34:50.774 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:34:50.774 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:34:50.774 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:50.774 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:50.774 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-osd.0.asok 2024-09-18T11:34:50.775 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: CEPH_ARGS= 2024-09-18T11:34:50.775 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.76995/ceph-osd.0.asok config get osd_max_backfills 2024-09-18T11:34:50.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-09-18T11:34:50.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:903: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-09-18T11:34:50.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd: teardown td/ceph-helpers 2024-09-18T11:34:50.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:34:50.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:34:50.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:34:50.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:34:50.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:34:50.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:34:50.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:34:50.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:34:50.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:34:50.959 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:34:50.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:34:50.960 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:34:50.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:34:50.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:34:50.962 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:34:50.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:34:50.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:34:50.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:34:50.964 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:34:50.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:34:50.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:34:50.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:34:50.970 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:34:50.970 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:50.970 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:50.971 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:34:50.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:34:50.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:34:50.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:34:50.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2024-09-18T11:34:50.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2024-09-18T11:34:50.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:911: test_activate_osd_after_mark_down: setup td/ceph-helpers 2024-09-18T11:34:50.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:34:50.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:34:50.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:34:50.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:34:50.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:34:50.979 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:34:50.979 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:34:50.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:34:50.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:34:50.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:34:50.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:34:50.980 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:34:50.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:34:50.980 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:34:50.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:34:50.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:34:50.981 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:34:50.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:34:50.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:34:50.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:34:50.981 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:34:50.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:34:50.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:34:50.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:34:50.987 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:34:50.987 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:50.987 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:50.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:34:50.988 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:34:50.988 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:34:50.988 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:34:50.988 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:34:50.988 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:50.988 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:50.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:34:50.989 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:34:50.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:34:50.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:34:50.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:34:50.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2024-09-18T11:34:50.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:34:50.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:34:50.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:34:50.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:34:50.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:34:50.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-18T11:34:51.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:34:51.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:34:51.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:34:51.346 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:34:51.346 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:51.346 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:51.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:34:51.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:34:51.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:34:51.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:34:51.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:34:51.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:34:51.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:34:51.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:34:51.598 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:34:51.599 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:34:51.599 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:34:51.599 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:34:51.599 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:51.599 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:51.599 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:34:51.599 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:34:51.599 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:34:51.675 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:34:51.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:34:51.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:34:51.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:34:51.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:34:51.676 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:34:51.676 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:34:51.676 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:34:51.676 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:34:51.676 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:51.676 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:51.676 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:34:51.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:34:51.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:34:51.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2024-09-18T11:34:51.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:34:51.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:34:51.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:34:51.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:34:51.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:34:51.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:34:52.601 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:34:52.601 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:34:52.602 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:34:52.602 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:34:52.602 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:52.602 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:52.602 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:34:52.602 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:34:52.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:34:52.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2024-09-18T11:34:52.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:34:52.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:34:52.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:34:52.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:34:52.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:34:52.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:34:52.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:34:52.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:34:52.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:34:52.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:34:52.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:34:52.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:34:52.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:34:52.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:34:52.630 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:34:52.630 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:34:52.630 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:34:52.630 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:34:52.630 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:52.630 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:34:52.630 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:34:52.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:34:52.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:34:52.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:34:52.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:34:52.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:34:52.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:34:52.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:34:52.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:34:52.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:34:52.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:34:52.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:34:52.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:34:52.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:34:52.631 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:34:52.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=64f97984-58e7-4b80-a309-b46a3e527abd 2024-09-18T11:34:52.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 64f97984-58e7-4b80-a309-b46a3e527abd' 2024-09-18T11:34:52.632 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 64f97984-58e7-4b80-a309-b46a3e527abd 2024-09-18T11:34:52.633 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:34:52.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDcuupm3R9SJhAAJ3QmQ/z4IEabRQJu3VDCTA== 2024-09-18T11:34:52.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDcuupm3R9SJhAAJ3QmQ/z4IEabRQJu3VDCTA=="}' 2024-09-18T11:34:52.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 64f97984-58e7-4b80-a309-b46a3e527abd -i td/ceph-helpers/0/new.json 2024-09-18T11:34:52.977 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:34:52.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:34:52.988 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDcuupm3R9SJhAAJ3QmQ/z4IEabRQJu3VDCTA== --osd-uuid 64f97984-58e7-4b80-a309-b46a3e527abd 2024-09-18T11:34:53.012 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:53.010+0000 7fea4f00d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:53.022 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:53.021+0000 7fea4f00d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:53.023 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:53.022+0000 7fea4f00d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:53.024 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:53.023+0000 7fea4f00d440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:34:53.024 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:53.023+0000 7fea4f00d440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:34:53.024 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:53.023+0000 7fea4f00d440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:34:53.063 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:53.061+0000 7fea4f00d440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:34:58.408 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:34:58.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:34:58.409 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:34:58.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:34:58.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:34:58.895 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:34:58.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:34:58.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:34:58.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:34:58.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:34:58.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:34:58.918 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:58.916+0000 7f6ec7c1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:58.920 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:58.918+0000 7f6ec7c1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:58.921 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:58.919+0000 7f6ec7c1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:59.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:34:59.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:34:59.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:34:59.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:34:59.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:34:59.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:59.249 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:34:59.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:34:59.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:59.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:59.446 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:34:59.445+0000 7f6ec7c1e440 -1 Falling back to public interface 2024-09-18T11:34:59.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:00.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:00.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:00.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:35:00.617 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:35:00.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:00.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:00.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:01.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:01.976 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:35:01.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:01.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:35:01.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:01.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:01.981 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:01.979+0000 7f6ec7c1e440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:35:02.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:03.340 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:03.340 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:03.340 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:35:03.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:35:03.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:03.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:03.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:04.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:04.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:04.792 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:35:04.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:35:04.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:04.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:05.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:06.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:06.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:06.152 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:35:06.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:35:06.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:06.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:06.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:07.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:07.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:07.731 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:35:07.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:35:07.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:07.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:08.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:09.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:09.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:09.112 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:35:09.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:35:09.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:09.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:09.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:10.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:10.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:10.476 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:35:10.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:35:10.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:10.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:10.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:11.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:11.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:11.992 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:35:11.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:35:11.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:11.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:12.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:13.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:13.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:13.352 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:35:13.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:35:13.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:13.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:14.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:15.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:15.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:15.350 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:35:15.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:35:15.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:15.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:15.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:16.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:16.874 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:35:16.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:16.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:35:16.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:16.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:17.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:18.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:18.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:18.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:35:18.577 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:35:18.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:18.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:19.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:20.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:20.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:20.194 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:35:20.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:35:20.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:20.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:20.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:21.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:21.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:21.716 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:35:21.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:35:21.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:21.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:22.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:23.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:23.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:23.286 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:35:23.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:35:23.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:23.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:23.898 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/80703,v1:127.0.0.1:6803/80703] [v2:127.0.0.1:6804/80703,v1:127.0.0.1:6805/80703] exists,up 64f97984-58e7-4b80-a309-b46a3e527abd 2024-09-18T11:35:23.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:35:23.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:35:23.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:35:23.899 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-09-18T11:35:23.899 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:35:23.899 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:35:23.900 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:35:23.900 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:23.900 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:23.900 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-osd.0.asok 2024-09-18T11:35:23.900 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-09-18T11:35:23.900 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.76995/ceph-osd.0.asok config get osd_max_backfills 2024-09-18T11:35:23.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2024-09-18T11:35:23.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2024-09-18T11:35:23.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2024-09-18T11:35:23.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2024-09-18T11:35:23.981 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:35:23.982 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:35:23.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:35:23.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:35:23.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:35:24.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:35:24.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: test_activate_osd_after_mark_down: ceph osd down 0 2024-09-18T11:35:24.683 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already down. 2024-09-18T11:35:24.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: test_activate_osd_after_mark_down: wait_for_osd down 0 2024-09-18T11:35:24.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-09-18T11:35:24.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:35:24.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:35:24.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:35:24.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:24.695 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:35:24.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:35:24.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:24.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-09-18T11:35:25.043 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/80703,v1:127.0.0.1:6803/80703] [v2:127.0.0.1:6804/80703,v1:127.0.0.1:6805/80703] exists 64f97984-58e7-4b80-a309-b46a3e527abd 2024-09-18T11:35:25.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:35:25.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:35:25.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:35:25.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-09-18T11:35:25.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-18T11:35:25.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:35:25.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T11:35:25.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:35:25.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:35:25.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:35:25.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:35:25.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:35:25.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:35:25.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:35:25.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:35:25.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:35:25.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:35:25.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:35:25.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:35:25.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:35:25.049 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:35:25.049 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:35:25.049 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:25.049 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:25.049 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:35:25.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:35:25.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:35:25.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:35:25.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:35:25.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:35:25.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:35:25.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:35:25.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:35:25.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:35:25.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-09-18T11:35:25.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:35:25.050 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:35:25.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T11:35:25.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-09-18T11:35:25.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-18T11:35:25.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T11:35:25.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:35:25.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:35:25.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:35:25.069 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:25.067+0000 7f746277f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:25.084 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:25.083+0000 7f746277f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:25.085 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:25.084+0000 7f746277f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:25.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T11:35:25.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:35:25.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:35:25.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:35:25.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:35:25.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:25.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:35:25.402 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:35:25.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:25.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:25.600 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:25.599+0000 7f746277f440 -1 Falling back to public interface 2024-09-18T11:35:25.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:26.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:26.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:26.761 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:35:26.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:35:26.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:26.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:27.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:28.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:28.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:28.121 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:35:28.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:35:28.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:28.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:28.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:28.559 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:28.557+0000 7f746277f440 -1 osd.0 5 log_to_monitors true 2024-09-18T11:35:29.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:29.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:29.487 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:35:29.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:35:29.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:29.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:29.862 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/81668,v1:127.0.0.1:6803/81668] [v2:127.0.0.1:6804/81668,v1:127.0.0.1:6805/81668] exists,up 64f97984-58e7-4b80-a309-b46a3e527abd 2024-09-18T11:35:29.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:35:29.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:35:29.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:35:29.864 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-09-18T11:35:29.864 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:35:29.864 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:35:29.864 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:35:29.864 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:29.864 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:29.864 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-osd.0.asok 2024-09-18T11:35:29.864 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-09-18T11:35:29.864 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.76995/ceph-osd.0.asok config get osd_max_backfills 2024-09-18T11:35:29.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2024-09-18T11:35:29.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-09-18T11:35:29.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2024-09-18T11:35:29.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:35:29.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:35:29.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:35:29.943 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:35:29.943 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:35:29.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:35:29.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:35:29.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:35:30.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:35:30.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:35:30.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:35:30.049 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:35:30.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:35:30.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:35:30.050 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:35:30.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:35:30.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:35:30.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:35:30.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:35:30.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:35:30.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:35:30.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:35:30.059 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:35:30.059 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:30.059 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:30.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:35:30.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:35:30.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:35:30.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:35:30.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2024-09-18T11:35:30.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:934: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2024-09-18T11:35:30.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2024-09-18T11:35:30.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:35:30.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:35:30.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:35:30.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:35:30.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:35:30.061 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:35:30.061 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:35:30.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:35:30.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:35:30.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:35:30.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:35:30.062 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:35:30.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:35:30.063 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:35:30.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:35:30.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:35:30.064 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:35:30.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:35:30.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:35:30.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:35:30.065 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:35:30.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:35:30.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:35:30.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:35:30.068 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:35:30.068 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:30.068 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:30.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:35:30.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:35:30.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:35:30.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:35:30.070 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:35:30.070 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:30.070 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:30.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:35:30.072 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:35:30.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:35:30.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:35:30.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:35:30.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2024-09-18T11:35:30.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:35:30.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:35:30.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:35:30.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:35:30.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:35:30.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-18T11:35:30.445 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:35:30.446 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:35:30.446 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:35:30.446 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:35:30.446 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:30.446 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:30.446 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:35:30.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:35:30.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:35:30.731 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:35:30.731 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:35:30.731 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:35:30.731 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:35:30.732 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:35:30.732 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:35:30.733 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:35:30.733 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:35:30.733 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:35:30.733 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:30.733 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:30.733 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:35:30.733 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:35:30.733 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:35:30.814 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:35:30.814 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:35:30.814 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:35:30.814 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:35:30.814 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:35:30.814 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:35:30.814 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:35:30.814 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:35:30.814 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:35:30.815 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:30.815 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:30.815 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:35:30.815 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:35:30.815 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:35:30.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:939: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2024-09-18T11:35:30.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:35:30.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:35:30.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:35:30.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:35:30.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:35:30.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:35:31.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:35:31.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:35:31.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:35:31.725 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:35:31.725 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:31.725 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:31.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:35:31.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:35:31.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:35:31.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-18T11:35:31.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:35:31.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:35:31.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:35:31.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:35:31.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:35:31.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:35:31.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:35:31.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:35:31.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:35:31.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:35:31.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:35:31.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:35:31.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:35:31.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:35:31.754 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:35:31.754 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:35:31.754 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:35:31.754 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:35:31.754 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:31.754 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:31.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:35:31.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:35:31.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:35:31.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:35:31.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:35:31.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:35:31.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:35:31.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:35:31.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:35:31.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:35:31.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:35:31.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:35:31.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-09-18T11:35:31.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:35:31.756 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:35:31.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4533bc2d-697e-48e7-a561-8b4292fa556a 2024-09-18T11:35:31.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4533bc2d-697e-48e7-a561-8b4292fa556a' 2024-09-18T11:35:31.756 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 4533bc2d-697e-48e7-a561-8b4292fa556a 2024-09-18T11:35:31.757 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:35:31.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQADu+pm+V2mLRAAQOznaN5RFFde9mpmTCX6Qg== 2024-09-18T11:35:31.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQADu+pm+V2mLRAAQOznaN5RFFde9mpmTCX6Qg=="}' 2024-09-18T11:35:31.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4533bc2d-697e-48e7-a561-8b4292fa556a -i td/ceph-helpers/0/new.json 2024-09-18T11:35:32.068 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:35:32.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:35:32.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQADu+pm+V2mLRAAQOznaN5RFFde9mpmTCX6Qg== --osd-uuid 4533bc2d-697e-48e7-a561-8b4292fa556a 2024-09-18T11:35:32.104 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:32.103+0000 7f23d640d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:32.114 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:32.113+0000 7f23d640d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:32.115 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:32.114+0000 7f23d640d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:32.116 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:32.114+0000 7f23d640d440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:35:32.116 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:32.114+0000 7f23d640d440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:35:32.116 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:32.114+0000 7f23d640d440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:35:32.146 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:32.145+0000 7f23d640d440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:35:37.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:35:37.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:35:37.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:35:37.494 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:35:37.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:35:37.980 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:35:37.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:35:37.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-18T11:35:37.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:35:37.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:35:37.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:35:38.004 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:38.002+0000 7efc3dc6e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:38.006 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:38.005+0000 7efc3dc6e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:38.007 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:38.006+0000 7efc3dc6e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:38.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:35:38.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:35:38.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:35:38.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:35:38.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:35:38.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:38.329 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:35:38.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:35:38.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:38.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:38.524 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:38.522+0000 7efc3dc6e440 -1 Falling back to public interface 2024-09-18T11:35:38.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:39.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:39.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:39.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:35:39.689 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:35:39.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:39.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:40.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:40.996 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:40.995+0000 7efc3dc6e440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:35:41.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:41.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:41.045 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:35:41.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:35:41.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:41.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:41.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:42.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:42.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:42.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:35:42.474 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:35:42.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:42.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:42.832 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/83342,v1:127.0.0.1:6803/83342] [v2:127.0.0.1:6804/83342,v1:127.0.0.1:6805/83342] exists,up 4533bc2d-697e-48e7-a561-8b4292fa556a 2024-09-18T11:35:42.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:35:42.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:35:42.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:35:42.836 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-18T11:35:42.836 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:35:42.836 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:35:42.837 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:35:42.837 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:42.837 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:42.837 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-osd.0.asok 2024-09-18T11:35:42.837 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-18T11:35:42.837 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.76995/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-09-18T11:35:42.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-18T11:35:42.915 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-18T11:35:42.915 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:35:42.915 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:35:42.915 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:35:42.915 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:42.915 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:42.915 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-osd.0.asok 2024-09-18T11:35:42.915 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-18T11:35:42.915 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.76995/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-09-18T11:35:42.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-18T11:35:42.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2024-09-18T11:35:42.995 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:35:42.995 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:35:42.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:35:42.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:35:42.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:35:44.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:35:44.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:950: test_activate_osd_skip_benchmark: ceph osd down 0 2024-09-18T11:35:44.704 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already down. 2024-09-18T11:35:44.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: wait_for_osd down 0 2024-09-18T11:35:44.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-09-18T11:35:44.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:35:44.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:35:44.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:35:44.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:44.715 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:35:44.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:35:44.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:44.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-09-18T11:35:45.069 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/83342,v1:127.0.0.1:6803/83342] [v2:127.0.0.1:6804/83342,v1:127.0.0.1:6805/83342] exists 4533bc2d-697e-48e7-a561-8b4292fa556a 2024-09-18T11:35:45.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:35:45.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:35:45.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:35:45.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:955: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-18T11:35:45.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-18T11:35:45.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:35:45.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T11:35:45.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:35:45.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:35:45.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:35:45.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:35:45.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:35:45.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:35:45.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:35:45.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:35:45.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:35:45.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:35:45.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:35:45.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:35:45.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:35:45.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:35:45.074 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:35:45.074 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:45.074 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:45.074 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:35:45.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:35:45.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:35:45.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:35:45.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:35:45.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:35:45.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:35:45.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:35:45.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:35:45.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:35:45.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-09-18T11:35:45.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:35:45.075 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:35:45.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T11:35:45.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-18T11:35:45.076 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-18T11:35:45.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T11:35:45.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:35:45.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:35:45.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:35:45.095 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:45.094+0000 7f7411729440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:45.105 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:45.104+0000 7f7411729440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:45.106 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:45.105+0000 7f7411729440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:45.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T11:35:45.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:35:45.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:35:45.432 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:35:45.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:35:45.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:35:45.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:45.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:35:45.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:45.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:45.626 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:45.625+0000 7f7411729440 -1 Falling back to public interface 2024-09-18T11:35:45.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:46.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:46.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:46.785 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:35:46.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:35:46.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:46.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:47.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:48.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:48.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:48.188 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:35:48.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:35:48.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:48.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:48.464 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:48.462+0000 7f7411729440 -1 osd.0 6 log_to_monitors true 2024-09-18T11:35:48.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:49.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:49.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:49.612 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:35:49.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:35:49.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:49.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:50.006 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/83984,v1:127.0.0.1:6803/83984] [v2:127.0.0.1:6804/83984,v1:127.0.0.1:6805/83984] exists,up 4533bc2d-697e-48e7-a561-8b4292fa556a 2024-09-18T11:35:50.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:35:50.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:35:50.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:35:50.007 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-18T11:35:50.007 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:35:50.008 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:35:50.008 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:35:50.008 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:50.008 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:50.008 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-osd.0.asok 2024-09-18T11:35:50.008 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-18T11:35:50.008 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.76995/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-09-18T11:35:50.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-18T11:35:50.088 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-18T11:35:50.089 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:35:50.089 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:35:50.089 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:35:50.089 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:50.089 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:50.089 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-osd.0.asok 2024-09-18T11:35:50.089 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-18T11:35:50.089 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.76995/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-09-18T11:35:50.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-18T11:35:50.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-18T11:35:50.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-18T11:35:50.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2024-09-18T11:35:50.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:35:50.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:35:50.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:35:50.168 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:35:50.169 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:35:50.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:35:50.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:35:50.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:35:50.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:35:50.273 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:35:50.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:35:50.274 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:35:50.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:35:50.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:35:50.276 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:35:50.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:35:50.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:35:50.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:35:50.277 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:35:50.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:35:50.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:35:50.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:35:50.282 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:35:50.282 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:50.282 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:50.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:35:50.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:35:50.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:35:50.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:35:50.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_destroy_osd td/ceph-helpers 2024-09-18T11:35:50.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:796: test_destroy_osd: local dir=td/ceph-helpers 2024-09-18T11:35:50.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:798: test_destroy_osd: setup td/ceph-helpers 2024-09-18T11:35:50.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:35:50.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:35:50.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:35:50.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:35:50.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:35:50.285 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:35:50.285 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:35:50.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:35:50.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:35:50.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:35:50.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:35:50.286 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:35:50.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:35:50.286 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:35:50.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:35:50.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:35:50.287 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:35:50.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:35:50.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:35:50.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:35:50.289 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:35:50.289 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:35:50.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:35:50.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:35:50.291 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:35:50.291 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:50.291 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:50.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:35:50.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:35:50.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:35:50.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:35:50.293 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:35:50.293 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:50.293 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:50.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:35:50.295 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:35:50.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:35:50.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:35:50.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:35:50.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:799: test_destroy_osd: run_mon td/ceph-helpers a 2024-09-18T11:35:50.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:35:50.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:35:50.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:35:50.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:35:50.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:35:50.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-18T11:35:50.651 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:35:50.651 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:35:50.651 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:35:50.651 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:35:50.651 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:50.652 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:50.652 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:35:50.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:35:50.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:35:50.912 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:35:50.912 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:35:50.912 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:35:50.912 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:35:50.912 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:35:50.912 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:35:50.913 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:35:50.913 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:35:50.914 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:35:50.914 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:50.914 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:50.914 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:35:50.914 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:35:50.914 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:35:50.990 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:35:50.990 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:35:50.990 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:35:50.990 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:35:50.990 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:35:50.990 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:35:50.990 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:35:50.990 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:35:50.990 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:35:50.990 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:50.990 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:50.990 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:35:50.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:35:50.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:35:51.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:800: test_destroy_osd: run_mgr td/ceph-helpers x 2024-09-18T11:35:51.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:35:51.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:35:51.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:35:51.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:35:51.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:35:51.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:35:51.906 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:35:51.906 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:35:51.906 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:35:51.906 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:35:51.906 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:51.907 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:51.907 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:35:51.907 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:35:51.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:35:51.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:801: test_destroy_osd: run_osd td/ceph-helpers 0 2024-09-18T11:35:51.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:35:51.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:35:51.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:35:51.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:35:51.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:35:51.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:35:51.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:35:51.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:35:51.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:35:51.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:35:51.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:35:51.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:35:51.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:35:51.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:35:51.936 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:35:51.936 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:35:51.936 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:35:51.936 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:35:51.936 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:51.936 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:35:51.936 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:35:51.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:35:51.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:35:51.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:35:51.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:35:51.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:35:51.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:35:51.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:35:51.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:35:51.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:35:51.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:35:51.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:35:51.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:35:51.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:35:51.937 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:35:51.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=353bde54-2110-45fc-a742-3acbe40619bd 2024-09-18T11:35:51.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 353bde54-2110-45fc-a742-3acbe40619bd' 2024-09-18T11:35:51.938 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 353bde54-2110-45fc-a742-3acbe40619bd 2024-09-18T11:35:51.938 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:35:51.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAXu+pm4p5+OBAAy8/3FUdTxlyHy+RhxwovOw== 2024-09-18T11:35:51.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAXu+pm4p5+OBAAy8/3FUdTxlyHy+RhxwovOw=="}' 2024-09-18T11:35:51.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 353bde54-2110-45fc-a742-3acbe40619bd -i td/ceph-helpers/0/new.json 2024-09-18T11:35:52.267 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:35:52.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:35:52.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAXu+pm4p5+OBAAy8/3FUdTxlyHy+RhxwovOw== --osd-uuid 353bde54-2110-45fc-a742-3acbe40619bd 2024-09-18T11:35:52.302 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:52.300+0000 7fafa031d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:52.313 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:52.311+0000 7fafa031d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:52.314 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:52.312+0000 7fafa031d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:52.314 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:52.313+0000 7fafa031d440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:35:52.314 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:52.313+0000 7fafa031d440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:35:52.315 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:52.313+0000 7fafa031d440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:35:52.353 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:52.351+0000 7fafa031d440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:35:57.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:35:57.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:35:57.724 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:35:57.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:35:57.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:35:58.204 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:35:58.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:35:58.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:35:58.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:35:58.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:35:58.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:35:58.227 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:58.226+0000 7f5848208440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:58.229 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:58.228+0000 7f5848208440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:58.230 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:58.229+0000 7f5848208440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:58.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:35:58.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:35:58.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:35:58.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:35:58.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:35:58.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:58.553 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:35:58.553 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:35:58.553 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:58.553 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:58.747 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:35:58.746+0000 7f5848208440 -1 Falling back to public interface 2024-09-18T11:35:58.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:59.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:59.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:59.907 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:35:59.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:35:59.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:59.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:00.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:01.220 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:36:01.218+0000 7f5848208440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:36:01.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:01.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:01.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:36:01.317 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:36:01.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:01.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:01.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:02.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:02.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:02.781 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:36:02.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:36:02.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:02.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:03.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:04.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:04.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:04.165 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:36:04.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:36:04.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:04.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:04.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:05.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:05.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:05.542 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:36:05.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:36:05.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:05.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:06.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:07.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:07.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:07.038 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:36:07.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:36:07.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:07.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:07.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:08.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:08.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:08.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:36:08.411 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:36:08.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:08.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:08.916 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:09.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:09.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:09.918 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:36:09.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:36:09.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:09.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:10.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:11.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:11.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:11.297 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:36:11.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:36:11.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:11.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:11.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:12.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:12.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:12.685 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:36:12.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:36:12.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:12.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:13.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:14.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:14.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:14.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:36:14.632 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:36:14.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:14.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:15.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:16.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:16.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:16.129 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:36:16.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:36:16.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:16.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:16.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:17.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:17.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:17.835 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:36:17.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:36:17.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:17.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:18.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:19.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:19.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:19.686 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:36:19.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:36:19.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:19.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:20.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:21.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:21.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:21.512 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:36:21.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:36:21.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:21.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:21.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:22.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:22.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:22.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:36:22.959 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:36:22.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:22.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:23.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:24.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:24.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:24.402 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:36:24.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:36:24.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:24.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:24.802 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/85661,v1:127.0.0.1:6803/85661] [v2:127.0.0.1:6804/85661,v1:127.0.0.1:6805/85661] exists,up 353bde54-2110-45fc-a742-3acbe40619bd 2024-09-18T11:36:24.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:36:24.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:36:24.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:36:24.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:802: test_destroy_osd: destroy_osd td/ceph-helpers 0 2024-09-18T11:36:24.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/ceph-helpers 2024-09-18T11:36:24.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=0 2024-09-18T11:36:24.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.0 2024-09-18T11:36:25.357 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already out. 2024-09-18T11:36:25.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-09-18T11:36:25.368 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:36:25.368 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:36:25.368 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:36:25.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:36:25.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:36:25.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:36:25.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.0 2024-09-18T11:36:26.031 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already down. 2024-09-18T11:36:26.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2024-09-18T11:36:26.464 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 does not exist 2024-09-18T11:36:26.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/ceph-helpers/0 2024-09-18T11:36:26.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers/0 2024-09-18T11:36:26.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:36:26.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers/0 KILL 2024-09-18T11:36:26.476 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:36:26.476 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:36:26.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:36:26.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:36:26.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:36:26.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:36:26.478 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:36:26.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:36:26.479 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:36:26.480 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:36:26.480 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:36:26.480 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:36:26.481 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:26.481 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:36:26.481 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:36:26.481 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:26.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:36:26.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:36:26.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers/0 2024-09-18T11:36:26.487 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:36:26.487 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:26.487 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:26.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:36:26.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:36:26.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:36:26.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/ceph-helpers/0 2024-09-18T11:36:26.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: ceph osd dump 2024-09-18T11:36:26.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: grep 'osd. ' 2024-09-18T11:36:26.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:804: test_destroy_osd: teardown td/ceph-helpers 2024-09-18T11:36:26.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:36:26.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:36:26.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:36:26.835 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:36:26.835 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:36:26.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:36:26.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:36:26.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:36:26.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:36:26.940 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:36:26.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:36:26.941 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:36:26.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:36:26.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:36:26.942 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:36:26.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:26.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:36:26.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:36:26.944 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:26.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:36:26.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:36:26.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:36:26.949 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:36:26.949 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:26.949 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:26.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:36:26.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:36:26.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:36:26.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:36:26.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_display_logs td/ceph-helpers 2024-09-18T11:36:26.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: test_display_logs: local dir=td/ceph-helpers 2024-09-18T11:36:26.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: test_display_logs: setup td/ceph-helpers 2024-09-18T11:36:26.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:36:26.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:36:26.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:36:26.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:36:26.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:36:26.952 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:36:26.952 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:36:26.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:36:26.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:36:26.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:36:26.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:36:26.953 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:36:26.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:36:26.954 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:36:26.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:36:26.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:36:26.958 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:36:26.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:26.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:36:26.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:36:26.958 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:26.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:36:26.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:36:26.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:36:26.963 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:36:26.963 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:26.964 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:26.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:36:26.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:36:26.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:36:26.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:36:26.964 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:36:26.965 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:26.965 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:26.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:36:26.966 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:36:26.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:36:26.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:36:26.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:36:26.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: test_display_logs: run_mon td/ceph-helpers a 2024-09-18T11:36:26.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:36:26.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:36:26.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:36:26.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:36:26.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:36:26.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-18T11:36:27.334 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:36:27.334 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:36:27.334 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:36:27.334 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:36:27.335 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:27.335 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:27.335 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:36:27.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:36:27.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:36:27.587 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:36:27.587 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:36:27.587 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:36:27.587 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:36:27.587 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:36:27.587 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:36:27.588 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:36:27.588 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:36:27.588 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:36:27.588 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:27.588 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:27.588 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:36:27.588 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:36:27.588 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:36:27.663 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:36:27.663 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:36:27.663 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:36:27.663 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:36:27.663 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:36:27.663 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:36:27.663 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:36:27.663 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:36:27.664 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:36:27.664 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:27.664 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:27.664 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:36:27.664 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:36:27.664 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:36:27.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: test_display_logs: kill_daemons td/ceph-helpers 2024-09-18T11:36:27.736 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:36:27.736 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:36:27.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:36:27.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:36:27.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:36:29.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:36:29.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: test_display_logs: display_logs td/ceph-helpers 2024-09-18T11:36:29.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: display_logs: local dir=td/ceph-helpers 2024-09-18T11:36:29.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2024-09-18T11:36:29.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: display_logs: read file 2024-09-18T11:36:29.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2024-09-18T11:36:29.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: display_logs: cat td/ceph-helpers/mon.a.log 2024-09-18T11:36:29.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: display_logs: read file 2024-09-18T11:36:29.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2024-09-18T11:36:29.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: test_display_logs: teardown td/ceph-helpers 2024-09-18T11:36:29.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:36:29.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:36:29.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:36:29.049 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:36:29.049 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:36:29.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:36:29.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:36:29.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:36:29.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:36:29.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:36:29.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:36:29.052 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:36:29.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:36:29.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:36:29.053 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:36:29.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:29.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:36:29.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:36:29.055 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:29.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:36:29.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:36:29.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:36:29.059 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:36:29.059 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:29.059 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:29.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:36:29.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:36:29.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:36:29.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:36:29.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2024-09-18T11:36:29.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2044: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2024-09-18T11:36:29.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2046: test_erasure_code_plugin_exists: setup td/ceph-helpers 2024-09-18T11:36:29.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:36:29.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:36:29.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:36:29.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:36:29.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:36:29.062 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:36:29.062 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:36:29.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:36:29.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:36:29.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:36:29.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:36:29.063 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:36:29.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:36:29.064 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:36:29.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:36:29.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:36:29.065 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:36:29.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:29.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:36:29.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:36:29.067 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:29.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:36:29.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:36:29.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:36:29.070 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:36:29.070 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:29.070 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:29.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:36:29.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:36:29.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:36:29.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:36:29.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:36:29.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:29.074 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:29.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:36:29.076 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:36:29.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:36:29.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:36:29.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:36:29.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2047: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2024-09-18T11:36:29.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:36:29.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:36:29.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:36:29.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:36:29.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:36:29.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-18T11:36:29.440 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:36:29.441 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:36:29.441 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:36:29.441 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:36:29.441 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:29.441 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:29.441 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:36:29.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:36:29.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:36:29.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:36:29.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:36:29.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:36:29.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:36:29.714 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:36:29.714 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:36:29.714 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:36:29.714 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:36:29.714 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:36:29.714 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:29.714 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:29.715 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:36:29.715 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:36:29.715 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:36:29.795 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:36:29.795 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:36:29.796 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:36:29.796 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:36:29.796 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:36:29.796 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:36:29.797 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:36:29.797 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:36:29.797 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:36:29.797 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:29.797 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:29.797 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:36:29.797 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:36:29.798 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:36:29.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2024-09-18T11:36:29.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:36:29.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:36:29.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:36:29.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:36:29.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:36:29.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:36:30.703 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:36:30.703 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:36:30.703 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:36:30.703 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:36:30.704 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:30.704 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:30.704 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:36:30.704 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:36:30.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:36:30.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2049: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2024-09-18T11:36:30.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: erasure_code_plugin_exists: local plugin=jerasure 2024-09-18T11:36:30.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: erasure_code_plugin_exists: local status 2024-09-18T11:36:30.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2024: erasure_code_plugin_exists: local grepstr 2024-09-18T11:36:30.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: erasure_code_plugin_exists: local s 2024-09-18T11:36:30.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: erasure_code_plugin_exists: case `uname` in 2024-09-18T11:36:30.730 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: erasure_code_plugin_exists: uname 2024-09-18T11:36:30.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2024-09-18T11:36:30.730 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2024-09-18T11:36:31.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: erasure_code_plugin_exists: s= 2024-09-18T11:36:31.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: erasure_code_plugin_exists: local status=0 2024-09-18T11:36:31.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-09-18T11:36:31.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-09-18T11:36:31.369 INFO:tasks.workunit.client.0.smithi157.stderr:erasure-code-profile TESTPROFILE does not exist 2024-09-18T11:36:31.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: erasure_code_plugin_exists: return 0 2024-09-18T11:36:31.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2024-09-18T11:36:31.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: erasure_code_plugin_exists: local plugin=FAKE 2024-09-18T11:36:31.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: erasure_code_plugin_exists: local status 2024-09-18T11:36:31.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2024: erasure_code_plugin_exists: local grepstr 2024-09-18T11:36:31.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: erasure_code_plugin_exists: local s 2024-09-18T11:36:31.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: erasure_code_plugin_exists: case `uname` in 2024-09-18T11:36:31.380 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: erasure_code_plugin_exists: uname 2024-09-18T11:36:31.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2024-09-18T11:36:31.380 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2024-09-18T11:36:31.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so): /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2024-09-18T11:36:31.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: erasure_code_plugin_exists: local status=5 2024-09-18T11:36:31.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2024-09-18T11:36:31.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2035: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so):' /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2024-09-18T11:36:31.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2035: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2024-09-18T11:36:31.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: erasure_code_plugin_exists: return 5 2024-09-18T11:36:31.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2024-09-18T11:36:31.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:36:31.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:36:31.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:36:31.530 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:36:31.530 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:36:31.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:36:31.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:36:31.531 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:36:31.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:36:31.635 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:36:31.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:36:31.636 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:36:31.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:36:31.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:36:31.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:36:31.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:31.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:36:31.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:36:31.638 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:31.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:36:31.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:36:31.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:36:31.641 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:36:31.642 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:31.642 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:31.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:36:31.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:36:31.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:36:31.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:36:31.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_expect_failure td/ceph-helpers 2024-09-18T11:36:31.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: test_expect_failure: local dir=td/ceph-helpers 2024-09-18T11:36:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1956: test_expect_failure: setup td/ceph-helpers 2024-09-18T11:36:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:36:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:36:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:36:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:36:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:36:31.645 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:36:31.645 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:36:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:36:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:36:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:36:31.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:36:31.646 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:36:31.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:36:31.646 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:36:31.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:36:31.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:36:31.647 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:36:31.648 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:31.648 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:36:31.648 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:36:31.648 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:31.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:36:31.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:36:31.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:36:31.651 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:36:31.651 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:31.651 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:31.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:36:31.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:36:31.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:36:31.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:36:31.653 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:36:31.653 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:31.653 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:31.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:36:31.655 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:36:31.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:36:31.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:36:31.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:36:31.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1957: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2024-09-18T11:36:31.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/ceph-helpers 2024-09-18T11:36:31.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-18T11:36:31.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local expected=FAIL 2024-09-18T11:36:31.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-18T11:36:31.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-18T11:36:31.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: bash -c 'echo FAIL ; exit 1' 2024-09-18T11:36:31.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-18T11:36:31.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-18T11:36:31.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-18T11:36:31.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: expect_failure: return 0 2024-09-18T11:36:31.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1959: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2024-09-18T11:36:31.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/ceph-helpers 2024-09-18T11:36:31.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-18T11:36:31.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local expected=FAIL 2024-09-18T11:36:31.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-18T11:36:31.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-18T11:36:31.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: bash -c 'echo FAIL ; exit 0' 2024-09-18T11:36:31.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1940: expect_failure: success=true 2024-09-18T11:36:31.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: true 2024-09-18T11:36:31.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: expect_failure: cat td/ceph-helpers/out 2024-09-18T11:36:31.659 INFO:tasks.workunit.client.0.smithi157.stderr:FAIL 2024-09-18T11:36:31.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: expect_failure: return 1 2024-09-18T11:36:31.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1960: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-18T11:36:31.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2024-09-18T11:36:31.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/ceph-helpers 2024-09-18T11:36:31.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-18T11:36:31.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local expected=FAIL 2024-09-18T11:36:31.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-18T11:36:31.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-18T11:36:31.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2024-09-18T11:36:31.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-18T11:36:31.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-18T11:36:31.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-18T11:36:31.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: expect_failure: cat td/ceph-helpers/out 2024-09-18T11:36:31.663 INFO:tasks.workunit.client.0.smithi157.stderr:UNEXPECTED 2024-09-18T11:36:31.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: expect_failure: return 1 2024-09-18T11:36:31.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-18T11:36:31.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: test_expect_failure: teardown td/ceph-helpers 2024-09-18T11:36:31.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:36:31.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:36:31.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:36:31.665 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:36:31.665 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:36:31.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:36:31.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:36:31.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:36:31.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:36:31.666 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:36:31.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:36:31.667 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:36:31.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:36:31.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:36:31.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:36:31.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:31.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:36:31.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:36:31.670 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:31.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:36:31.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:36:31.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:36:31.672 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:36:31.672 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:31.673 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:31.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:36:31.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:36:31.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:36:31.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:36:31.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_flush_pg_stats td/ceph-helpers 2024-09-18T11:36:31.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2196: test_flush_pg_stats: local dir=td/ceph-helpers 2024-09-18T11:36:31.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2198: test_flush_pg_stats: setup td/ceph-helpers 2024-09-18T11:36:31.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:36:31.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:36:31.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:36:31.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:36:31.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:36:31.677 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:36:31.677 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:36:31.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:36:31.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:36:31.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:36:31.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:36:31.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:36:31.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:36:31.679 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:36:31.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:36:31.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:36:31.679 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:36:31.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:31.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:36:31.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:36:31.679 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:36:31.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:36:31.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:36:31.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:36:31.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:36:31.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:31.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:31.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:36:31.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:36:31.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:36:31.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:36:31.683 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:36:31.684 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:31.684 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:31.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:36:31.685 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:36:31.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:36:31.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:36:31.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:36:31.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:36:31.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:36:31.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:36:31.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:36:31.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:36:31.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:36:31.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:36:32.034 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:36:32.034 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:36:32.034 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:36:32.034 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:36:32.034 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:32.034 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:32.034 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:36:32.034 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:36:32.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:36:32.289 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:36:32.289 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:36:32.289 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:36:32.289 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:36:32.289 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:36:32.289 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:36:32.289 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:36:32.290 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:36:32.290 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:36:32.290 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:32.290 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:32.290 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:36:32.290 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:36:32.290 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:36:32.361 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:36:32.361 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:36:32.361 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:36:32.361 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:36:32.362 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:36:32.362 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:36:32.362 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:36:32.362 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:36:32.362 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:36:32.362 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:32.362 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:32.362 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:36:32.362 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:36:32.362 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:36:32.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: test_flush_pg_stats: run_mgr td/ceph-helpers x 2024-09-18T11:36:32.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:36:32.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:36:32.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:36:32.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:36:32.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:36:32.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:36:33.271 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:36:33.271 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:36:33.272 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:36:33.272 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:36:33.272 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:33.272 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:33.272 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:36:33.272 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:36:33.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:36:33.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: test_flush_pg_stats: run_osd td/ceph-helpers 0 2024-09-18T11:36:33.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:36:33.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:36:33.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:36:33.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:36:33.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:36:33.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:36:33.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:36:33.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:36:33.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:36:33.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:36:33.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:36:33.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:36:33.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:36:33.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:36:33.302 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:36:33.302 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:36:33.302 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:36:33.302 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:36:33.302 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:33.302 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:36:33.302 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:36:33.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:36:33.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:36:33.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:36:33.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:36:33.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:36:33.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:36:33.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:36:33.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:36:33.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:36:33.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:36:33.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:36:33.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:36:33.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:36:33.304 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:36:33.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6733188c-63cd-413d-8eb9-cd13e9ccf8f1 2024-09-18T11:36:33.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6733188c-63cd-413d-8eb9-cd13e9ccf8f1' 2024-09-18T11:36:33.304 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 6733188c-63cd-413d-8eb9-cd13e9ccf8f1 2024-09-18T11:36:33.304 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:36:33.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBu+pmgbu1EhAAt6EytBr9ji/Lql5p4oSbKQ== 2024-09-18T11:36:33.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBu+pmgbu1EhAAt6EytBr9ji/Lql5p4oSbKQ=="}' 2024-09-18T11:36:33.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6733188c-63cd-413d-8eb9-cd13e9ccf8f1 -i td/ceph-helpers/0/new.json 2024-09-18T11:36:33.640 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:36:33.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:36:33.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBu+pmgbu1EhAAt6EytBr9ji/Lql5p4oSbKQ== --osd-uuid 6733188c-63cd-413d-8eb9-cd13e9ccf8f1 2024-09-18T11:36:33.674 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:36:33.673+0000 7f2a2e929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:36:33.719 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:36:33.718+0000 7f2a2e929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:36:33.720 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:36:33.719+0000 7f2a2e929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:36:33.721 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:36:33.719+0000 7f2a2e929440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:36:33.721 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:36:33.719+0000 7f2a2e929440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:36:33.721 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:36:33.719+0000 7f2a2e929440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:36:33.759 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:36:33.758+0000 7f2a2e929440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:36:39.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:36:39.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:36:39.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:36:39.109 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:36:39.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:36:39.600 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:36:39.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:36:39.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:36:39.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:36:39.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:36:39.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:36:39.622 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:36:39.620+0000 7f1a9c023440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:36:39.624 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:36:39.623+0000 7f1a9c023440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:36:39.625 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:36:39.624+0000 7f1a9c023440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:36:39.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:36:39.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:36:39.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:36:39.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:36:39.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:36:39.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:39.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:36:39.955 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:36:39.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:39.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:40.139 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:36:40.138+0000 7f1a9c023440 -1 Falling back to public interface 2024-09-18T11:36:40.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:41.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:41.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:41.322 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:36:41.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:36:41.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:41.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:41.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:42.660 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:36:42.659+0000 7f1a9c023440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:36:42.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:42.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:42.686 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:36:42.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:36:42.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:42.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:43.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:44.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:44.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:44.047 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:36:44.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:36:44.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:44.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:44.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:45.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:45.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:45.457 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:36:45.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:36:45.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:45.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:45.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:46.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:46.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:46.835 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:36:46.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:36:46.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:46.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:47.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:48.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:48.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:48.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:36:48.312 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:36:48.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:48.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:48.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:49.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:49.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:49.733 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:36:49.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:36:49.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:49.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:50.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:51.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:51.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:51.093 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:36:51.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:36:51.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:51.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:51.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:52.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:52.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:52.496 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:36:52.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:36:52.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:52.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:52.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:53.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:53.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:53.854 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:36:53.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:36:53.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:53.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:54.289 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:55.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:55.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:55.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:36:55.291 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:36:55.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:55.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:55.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:56.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:56.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:56.740 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:36:56.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:36:56.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:56.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:57.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:58.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:58.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:58.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:36:58.138 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:36:58.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:58.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:36:58.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:59.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:59.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:59.876 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:36:59.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:36:59.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:59.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:00.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:01.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:01.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:01.728 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:37:01.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:37:01.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:01.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:02.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:03.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:03.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:03.402 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:37:03.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:37:03.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:03.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:04.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:05.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:05.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:05.345 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:37:05.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:37:05.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:05.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:05.721 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/88205,v1:127.0.0.1:6803/88205] [v2:127.0.0.1:6804/88205,v1:127.0.0.1:6805/88205] exists,up 6733188c-63cd-413d-8eb9-cd13e9ccf8f1 2024-09-18T11:37:05.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:37:05.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:37:05.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:37:05.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: test_flush_pg_stats: create_rbd_pool 2024-09-18T11:37:05.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:37:06.075 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:37:06.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:37:06.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:37:06.620 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:37:06.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:37:07.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:37:08.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: test_flush_pg_stats: rados -p rbd put obj /etc/group 2024-09-18T11:37:08.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2204: test_flush_pg_stats: flush_pg_stats 2024-09-18T11:37:08.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:37:08.784 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:37:09.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T11:37:09.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:37:09.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:37:09.175 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:37:09.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836485 2024-09-18T11:37:09.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836485 2024-09-18T11:37:09.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485' 2024-09-18T11:37:09.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:37:09.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836485 2024-09-18T11:37:09.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:37:09.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:37:09.284 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836485 2024-09-18T11:37:09.284 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:37:09.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836485 2024-09-18T11:37:09.285 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836485 2024-09-18T11:37:09.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-18T11:37:09.286 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:37:09.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836483 -lt 21474836485 2024-09-18T11:37:09.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:37:10.706 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:37:10.707 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:37:11.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836486 -lt 21474836485 2024-09-18T11:37:11.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2024-09-18T11:37:11.072 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2206: test_flush_pg_stats: ceph df detail --format=json 2024-09-18T11:37:11.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2206: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2024-09-18T11:37:11.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2206: test_flush_pg_stats: stored=6798 2024-09-18T11:37:11.585 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: test_flush_pg_stats: ceph df detail --format=json 2024-09-18T11:37:11.585 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2024-09-18T11:37:12.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: test_flush_pg_stats: stored_raw=6798 2024-09-18T11:37:12.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: test_flush_pg_stats: test 6798 -gt 0 2024-09-18T11:37:12.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: test_flush_pg_stats: test 6798 == 6798 2024-09-18T11:37:12.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: test_flush_pg_stats: teardown td/ceph-helpers 2024-09-18T11:37:12.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:37:12.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:37:12.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:37:12.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:37:12.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:37:12.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:37:12.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:37:12.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:37:12.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:37:12.200 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:37:12.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:37:12.201 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:37:12.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:37:12.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:37:12.202 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:37:12.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:37:12.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:37:12.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:37:12.203 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:37:12.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:37:12.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:37:12.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:37:12.210 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:37:12.210 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:12.210 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:12.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:37:12.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:37:12.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:37:12.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:37:12.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_config td/ceph-helpers 2024-09-18T11:37:12.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: local dir=td/ceph-helpers 2024-09-18T11:37:12.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: test_get_config: setup td/ceph-helpers 2024-09-18T11:37:12.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:37:12.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:37:12.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:37:12.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:37:12.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:37:12.213 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:37:12.213 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:37:12.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:37:12.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:37:12.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:37:12.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:37:12.214 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:37:12.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:37:12.214 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:37:12.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:37:12.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:37:12.216 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:37:12.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:37:12.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:37:12.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:37:12.217 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:37:12.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:37:12.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:37:12.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:37:12.219 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:37:12.220 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:12.220 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:12.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:37:12.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:37:12.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:37:12.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:37:12.222 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:37:12.222 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:12.222 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:12.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:37:12.224 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:37:12.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:37:12.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:37:12.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:37:12.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1140: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:37:12.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:37:12.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:37:12.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:37:12.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:37:12.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:37:12.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:37:12.588 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:37:12.588 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:37:12.588 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:37:12.588 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:37:12.588 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:12.588 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:12.589 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:37:12.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:37:12.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:37:12.824 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:37:12.824 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:37:12.824 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:37:12.824 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:37:12.824 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:37:12.824 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:37:12.825 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:37:12.825 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:37:12.825 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:37:12.825 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:12.825 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:12.825 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:37:12.825 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:37:12.825 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:37:12.902 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:37:12.902 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:37:12.902 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:37:12.902 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:37:12.903 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:37:12.903 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:37:12.903 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:37:12.903 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:37:12.903 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:37:12.903 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:12.903 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:12.903 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:37:12.903 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:37:12.903 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:37:12.978 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: get_config mon a osd_pool_default_size 2024-09-18T11:37:12.978 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:37:12.978 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:37:12.978 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_pool_default_size 2024-09-18T11:37:12.978 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_pool_default_size 2024-09-18T11:37:12.979 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:37:12.979 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:37:12.979 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:37:12.979 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:37:12.979 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:12.979 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:12.979 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:37:12.979 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:37:12.979 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get osd_pool_default_size 2024-09-18T11:37:13.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: test 1 = 1 2024-09-18T11:37:13.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: test_get_config: run_mgr td/ceph-helpers x 2024-09-18T11:37:13.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:37:13.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:37:13.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:37:13.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:37:13.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:37:13.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:37:13.893 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:37:13.893 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:37:13.893 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:37:13.893 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:37:13.893 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:13.893 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:13.894 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:37:13.894 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:37:13.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:37:13.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1143: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2024-09-18T11:37:13.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:37:13.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:37:13.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:37:13.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:37:13.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:37:13.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:37:13.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:37:13.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:37:13.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:37:13.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:37:13.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:37:13.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:37:13.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:37:13.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:37:13.922 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:37:13.922 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:37:13.922 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:37:13.923 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:37:13.923 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:13.923 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:13.923 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:37:13.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:37:13.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:37:13.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:37:13.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:37:13.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:37:13.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:37:13.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:37:13.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:37:13.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:37:13.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:37:13.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:37:13.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_max_scrubs=3 2024-09-18T11:37:13.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:37:13.924 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:37:13.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=42971c47-9041-4989-b5bf-eb691a52183e 2024-09-18T11:37:13.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 42971c47-9041-4989-b5bf-eb691a52183e' 2024-09-18T11:37:13.924 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 42971c47-9041-4989-b5bf-eb691a52183e 2024-09-18T11:37:13.925 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:37:13.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpu+pm8EW8NxAAMZEQ2pzclitB2I5NXBX3Iw== 2024-09-18T11:37:13.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpu+pm8EW8NxAAMZEQ2pzclitB2I5NXBX3Iw=="}' 2024-09-18T11:37:13.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 42971c47-9041-4989-b5bf-eb691a52183e -i td/ceph-helpers/0/new.json 2024-09-18T11:37:14.245 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:37:14.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:37:14.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQBpu+pm8EW8NxAAMZEQ2pzclitB2I5NXBX3Iw== --osd-uuid 42971c47-9041-4989-b5bf-eb691a52183e 2024-09-18T11:37:14.280 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:14.279+0000 7f04f603b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:14.291 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:14.290+0000 7f04f603b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:14.292 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:14.291+0000 7f04f603b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:14.293 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:14.291+0000 7f04f603b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:37:14.293 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:14.291+0000 7f04f603b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:37:14.293 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:14.291+0000 7f04f603b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:37:14.331 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:14.330+0000 7f04f603b440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:37:19.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:37:19.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:37:19.711 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:37:19.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:37:19.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:37:20.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:37:20.193 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:37:20.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2024-09-18T11:37:20.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:37:20.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:37:20.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:37:20.216 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:20.214+0000 7f606f022440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:20.218 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:20.216+0000 7f606f022440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:20.219 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:20.217+0000 7f606f022440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:20.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:37:20.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:37:20.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:37:20.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:37:20.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:37:20.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:20.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:37:20.543 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:37:20.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:20.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:20.734 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:20.732+0000 7f606f022440 -1 Falling back to public interface 2024-09-18T11:37:20.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:21.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:21.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:21.910 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:37:21.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:37:21.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:21.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:22.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:23.189 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:23.188+0000 7f606f022440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:37:23.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:23.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:23.271 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:37:23.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:37:23.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:23.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:23.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:24.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:24.793 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:37:24.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:24.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:37:24.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:24.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:25.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:26.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:26.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:26.204 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:37:26.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:37:26.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:26.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:26.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:27.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:27.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:27.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:37:27.594 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:37:27.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:27.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:28.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:29.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:29.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:29.069 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:37:29.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:37:29.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:29.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:29.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:30.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:30.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:30.442 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:37:30.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:37:30.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:30.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:30.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:31.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:31.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:31.949 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:37:31.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:37:31.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:31.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:32.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:33.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:33.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:33.358 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:37:33.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:37:33.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:33.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:33.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:34.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:34.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:34.734 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:37:34.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:37:34.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:34.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:35.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:36.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:36.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:36.491 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:37:36.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:37:36.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:36.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:36.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:37.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:37.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:37.983 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:37:37.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:37:37.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:37.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:38.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:39.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:39.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:39.826 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:37:39.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:37:39.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:39.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:40.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:41.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:41.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:41.327 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:37:41.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:37:41.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:41.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:42.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:43.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:43.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:43.013 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:37:43.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:37:43.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:43.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:43.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:44.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:44.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:44.676 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:37:44.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:37:44.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:44.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:45.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:46.371 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:46.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:46.372 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:37:46.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:37:46.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:46.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:46.802 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/90465,v1:127.0.0.1:6803/90465] [v2:127.0.0.1:6804/90465,v1:127.0.0.1:6805/90465] exists,up 42971c47-9041-4989-b5bf-eb691a52183e 2024-09-18T11:37:46.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:37:46.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:37:46.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:37:46.804 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: get_config osd 0 osd_max_scrubs 2024-09-18T11:37:46.804 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-09-18T11:37:46.804 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-09-18T11:37:46.804 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_max_scrubs 2024-09-18T11:37:46.804 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_max_scrubs 2024-09-18T11:37:46.804 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-09-18T11:37:46.804 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:37:46.804 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:37:46.805 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:37:46.805 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:46.805 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:46.805 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-osd.0.asok 2024-09-18T11:37:46.805 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:37:46.805 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-osd.0.asok config get osd_max_scrubs 2024-09-18T11:37:46.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: test 3 = 3 2024-09-18T11:37:46.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1145: test_get_config: teardown td/ceph-helpers 2024-09-18T11:37:46.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:37:46.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:37:46.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:37:46.884 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:37:46.884 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:37:46.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:37:46.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:37:46.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:37:46.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:37:46.989 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:37:46.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:37:46.990 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:37:46.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:37:46.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:37:46.992 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:37:46.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:37:46.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:37:46.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:37:46.993 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:37:46.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:37:46.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:37:46.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:37:47.000 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:37:47.000 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:47.000 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:47.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:37:47.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:37:47.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:37:47.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:37:47.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2024-09-18T11:37:47.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2024-09-18T11:37:47.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: test_get_is_making_recovery_progress: setup td/ceph-helpers 2024-09-18T11:37:47.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:37:47.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:37:47.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:37:47.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:37:47.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:37:47.003 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:37:47.003 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:37:47.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:37:47.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:37:47.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:37:47.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:37:47.004 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:37:47.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:37:47.004 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:37:47.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:37:47.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:37:47.005 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:37:47.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:37:47.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:37:47.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:37:47.006 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:37:47.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:37:47.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:37:47.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:37:47.009 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:37:47.009 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:47.009 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:47.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:37:47.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:37:47.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:37:47.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:37:47.011 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:37:47.011 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:47.011 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:47.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:37:47.013 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:37:47.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:37:47.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:37:47.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:37:47.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1347: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2024-09-18T11:37:47.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:37:47.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:37:47.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:37:47.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:37:47.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:37:47.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-18T11:37:47.381 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:37:47.381 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:37:47.382 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:37:47.382 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:37:47.382 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:47.382 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:47.382 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:37:47.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:37:47.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:37:47.641 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:37:47.641 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:37:47.641 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:37:47.641 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:37:47.641 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:37:47.641 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:37:47.642 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:37:47.642 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:37:47.642 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:37:47.642 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:47.643 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:47.643 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:37:47.643 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:37:47.643 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:37:47.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:37:47.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:37:47.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:37:47.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:37:47.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:37:47.720 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:37:47.721 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:37:47.721 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:37:47.721 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:37:47.721 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:47.721 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:47.721 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:37:47.721 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:37:47.721 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:37:47.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1348: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2024-09-18T11:37:47.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:37:47.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:37:47.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:37:47.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:37:47.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:37:47.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:37:48.653 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:37:48.653 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:37:48.653 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:37:48.653 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:37:48.654 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:48.654 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:48.654 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:37:48.654 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:37:48.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:37:48.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1349: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2024-09-18T11:37:48.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T11:37:48.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T11:37:48.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T11:37:48.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T11:37:48.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T11:37:48.679 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T11:37:48.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T11:37:48.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T11:37:48.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1350: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2024-09-18T11:37:48.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:37:48.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:37:48.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:37:48.929 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:37:48.929 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:37:48.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:37:48.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:37:48.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:37:49.033 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:37:49.034 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:37:49.034 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:37:49.034 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:37:49.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:37:49.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:37:49.036 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:37:49.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:37:49.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:37:49.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:37:49.037 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:37:49.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:37:49.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:37:49.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:37:49.040 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:37:49.040 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:49.040 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:49.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:37:49.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:37:49.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:37:49.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:37:49.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2024-09-18T11:37:49.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2024-09-18T11:37:49.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1535: test_get_last_scrub_stamp: setup td/ceph-helpers 2024-09-18T11:37:49.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:37:49.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:37:49.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:37:49.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:37:49.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:37:49.043 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:37:49.043 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:37:49.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:37:49.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:37:49.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:37:49.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:37:49.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:37:49.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:37:49.045 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:37:49.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:37:49.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:37:49.046 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:37:49.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:37:49.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:37:49.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:37:49.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:37:49.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:37:49.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:37:49.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:37:49.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:37:49.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:49.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:49.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:37:49.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:37:49.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:37:49.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:37:49.055 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:37:49.055 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:49.055 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:49.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:37:49.055 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:37:49.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:37:49.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:37:49.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:37:49.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1536: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:37:49.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:37:49.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:37:49.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:37:49.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:37:49.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:37:49.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:37:49.422 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:37:49.422 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:37:49.422 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:37:49.422 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:37:49.422 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:49.422 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:49.422 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:37:49.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:37:49.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:37:49.674 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:37:49.674 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:37:49.674 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:37:49.675 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:37:49.675 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:37:49.675 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:37:49.675 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:37:49.675 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:37:49.675 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:37:49.675 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:49.675 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:49.675 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:37:49.675 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:37:49.675 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:37:49.750 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:37:49.751 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:37:49.751 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:37:49.751 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:37:49.751 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:37:49.751 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:37:49.751 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:37:49.751 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:37:49.751 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:37:49.751 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:49.751 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:49.751 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:37:49.751 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:37:49.751 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:37:49.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1537: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2024-09-18T11:37:49.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:37:49.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:37:49.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:37:49.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:37:49.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:37:49.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:37:50.659 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:37:50.659 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:37:50.659 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:37:50.659 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:37:50.659 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:50.660 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:50.660 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:37:50.660 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:37:50.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:37:50.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1538: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2024-09-18T11:37:50.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:37:50.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:37:50.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:37:50.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:37:50.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:37:50.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:37:50.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:37:50.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:37:50.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:37:50.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:37:50.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:37:50.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:37:50.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:37:50.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:37:50.687 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:37:50.687 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:37:50.687 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:37:50.687 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:37:50.687 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:50.688 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:37:50.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:37:50.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:37:50.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:37:50.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:37:50.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:37:50.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:37:50.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:37:50.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:37:50.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:37:50.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:37:50.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:37:50.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:37:50.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:37:50.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:37:50.689 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:37:50.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a3822f29-5e98-4cc4-818a-8182f5ac943d 2024-09-18T11:37:50.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a3822f29-5e98-4cc4-818a-8182f5ac943d' 2024-09-18T11:37:50.690 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 a3822f29-5e98-4cc4-818a-8182f5ac943d 2024-09-18T11:37:50.690 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:37:50.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCOu+pma8HCKRAAOW/Z/gN7W9NYozQtcS7jGA== 2024-09-18T11:37:50.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCOu+pma8HCKRAAOW/Z/gN7W9NYozQtcS7jGA=="}' 2024-09-18T11:37:50.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a3822f29-5e98-4cc4-818a-8182f5ac943d -i td/ceph-helpers/0/new.json 2024-09-18T11:37:51.002 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:37:51.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:37:51.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCOu+pma8HCKRAAOW/Z/gN7W9NYozQtcS7jGA== --osd-uuid a3822f29-5e98-4cc4-818a-8182f5ac943d 2024-09-18T11:37:51.037 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:51.035+0000 7f3602231440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:51.049 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:51.048+0000 7f3602231440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:51.050 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:51.049+0000 7f3602231440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:51.051 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:51.049+0000 7f3602231440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:37:51.051 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:51.049+0000 7f3602231440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:37:51.051 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:51.049+0000 7f3602231440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:37:51.081 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:51.079+0000 7f3602231440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:37:56.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:37:56.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:37:56.411 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:37:56.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:37:56.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:37:56.895 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:37:56.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:37:56.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:37:56.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:37:56.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:37:56.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:37:56.917 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:56.916+0000 7ff534727440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:56.919 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:56.918+0000 7ff534727440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:56.920 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:56.919+0000 7ff534727440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:57.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:37:57.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:37:57.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:37:57.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:37:57.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:37:57.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:57.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:37:57.245 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:37:57.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:57.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:57.442 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:57.441+0000 7ff534727440 -1 Falling back to public interface 2024-09-18T11:37:57.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:58.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:58.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:58.600 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:37:58.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:37:58.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:58.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:58.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:59.931 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:37:59.930+0000 7ff534727440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:37:59.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:59.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:59.994 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:37:59.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:37:59.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:59.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:00.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:01.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:01.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:01.487 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:38:01.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:38:01.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:01.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:01.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:02.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:02.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:02.896 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:38:02.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:38:02.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:02.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:03.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:04.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:04.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:04.301 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:38:04.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:38:04.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:04.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:04.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:05.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:05.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:05.766 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:38:05.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:38:05.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:05.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:06.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:07.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:07.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:07.146 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:38:07.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:38:07.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:07.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:07.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:08.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:08.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:08.578 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:38:08.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:38:08.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:08.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:08.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:09.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:09.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:09.938 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:38:09.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:38:09.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:09.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:10.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:11.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:11.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:11.313 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:38:11.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:38:11.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:11.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:12.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:13.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:13.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:13.150 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:38:13.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:38:13.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:13.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:13.724 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:14.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:14.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:14.725 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:38:14.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:38:14.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:14.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:15.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:16.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:16.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:16.528 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:38:16.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:38:16.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:16.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:17.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:18.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:18.274 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:38:18.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:18.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:38:18.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:18.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:18.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:19.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:19.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:19.703 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:38:19.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:38:19.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:19.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:20.408 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:21.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:21.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:21.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:38:21.409 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:38:21.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:21.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:21.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:22.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:22.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:22.958 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:38:22.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:38:22.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:22.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:23.319 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/92688,v1:127.0.0.1:6803/92688] [v2:127.0.0.1:6804/92688,v1:127.0.0.1:6805/92688] exists,up a3822f29-5e98-4cc4-818a-8182f5ac943d 2024-09-18T11:38:23.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:38:23.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:38:23.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:38:23.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1539: test_get_last_scrub_stamp: create_rbd_pool 2024-09-18T11:38:23.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:38:23.667 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:38:23.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:38:23.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:38:24.149 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:38:24.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:38:25.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:38:26.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1540: test_get_last_scrub_stamp: wait_for_clean 2024-09-18T11:38:26.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:38:26.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:38:26.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:38:26.126 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:38:26.126 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:38:26.126 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:38:26.127 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:38:26.127 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:38:26.127 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:38:26.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:38:26.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:38:26.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:38:26.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:38:26.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:38:26.223 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:38:26.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T11:38:26.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:38:26.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:38:26.575 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:38:26.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836485 2024-09-18T11:38:26.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836485 2024-09-18T11:38:26.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485' 2024-09-18T11:38:26.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:38:26.683 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836485 2024-09-18T11:38:26.683 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:38:26.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:38:26.684 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836485 2024-09-18T11:38:26.684 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:38:26.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836485 2024-09-18T11:38:26.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-18T11:38:26.686 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836485 2024-09-18T11:38:26.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:38:27.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836483 -lt 21474836485 2024-09-18T11:38:27.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:38:28.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:38:28.037 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:38:28.408 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836486 -lt 21474836485 2024-09-18T11:38:28.408 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:38:28.408 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:38:28.408 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:38:28.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:38:28.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:38:28.880 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:38:28.880 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:38:28.880 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:38:28.880 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:38:28.881 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:38:28.881 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:38:29.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:38:29.257 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:38:29.257 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:38:29.257 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:38:29.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T11:38:29.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:38:29.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:38:29.733 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2024-09-18T11:38:29.733 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:38:29.733 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:38:29.734 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:38:29.734 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:38:30.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: stamp=2024-09-18T11:38:23.975006+0000 2024-09-18T11:38:30.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1542: test_get_last_scrub_stamp: test -n 2024-09-18T11:38:23.975006+0000 2024-09-18T11:38:30.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1543: test_get_last_scrub_stamp: teardown td/ceph-helpers 2024-09-18T11:38:30.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:38:30.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:38:30.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:38:30.074 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:38:30.075 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:38:30.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:38:30.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:38:30.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:38:30.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:38:30.180 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:38:30.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:38:30.180 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:38:30.181 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:38:30.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:38:30.182 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:38:30.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:38:30.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:38:30.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:38:30.183 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:38:30.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:38:30.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:38:30.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:38:30.190 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:38:30.190 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:38:30.190 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:38:30.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:38:30.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:38:30.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:38:30.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:38:30.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_not_primary td/ceph-helpers 2024-09-18T11:38:30.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local dir=td/ceph-helpers 2024-09-18T11:38:30.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: setup td/ceph-helpers 2024-09-18T11:38:30.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:38:30.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:38:30.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:38:30.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:38:30.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:38:30.193 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:38:30.193 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:38:30.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:38:30.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:38:30.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:38:30.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:38:30.194 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:38:30.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:38:30.195 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:38:30.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:38:30.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:38:30.196 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:38:30.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:38:30.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:38:30.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:38:30.197 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:38:30.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:38:30.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:38:30.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:38:30.200 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:38:30.200 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:38:30.200 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:38:30.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:38:30.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:38:30.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:38:30.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:38:30.202 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:38:30.202 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:38:30.202 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:38:30.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:38:30.204 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:38:30.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:38:30.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:38:30.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:38:30.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-18T11:38:30.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:38:30.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:38:30.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:38:30.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:38:30.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:38:30.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-18T11:38:30.557 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:38:30.558 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:38:30.558 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:38:30.558 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:38:30.558 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:38:30.558 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:38:30.558 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:38:30.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-18T11:38:30.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:38:30.834 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:38:30.834 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:38:30.834 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:38:30.834 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:38:30.834 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:38:30.835 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:38:30.836 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:38:30.836 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:38:30.836 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:38:30.836 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:38:30.836 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:38:30.836 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:38:30.836 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:38:30.836 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:38:30.910 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:38:30.912 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:38:30.912 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:38:30.912 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:38:30.912 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:38:30.912 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:38:30.912 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:38:30.912 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:38:30.913 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:38:30.913 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:38:30.913 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:38:30.913 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:38:30.913 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:38:30.913 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:38:30.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: run_mgr td/ceph-helpers x 2024-09-18T11:38:30.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:38:30.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:38:30.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:38:30.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:38:30.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:38:30.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:38:31.829 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:38:31.829 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:38:31.829 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:38:31.829 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:38:31.829 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:38:31.829 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:38:31.829 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:38:31.829 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:38:31.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:38:31.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1243: test_get_not_primary: run_osd td/ceph-helpers 0 2024-09-18T11:38:31.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:38:31.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:38:31.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:38:31.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:38:31.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:38:31.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:38:31.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:38:31.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:38:31.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:38:31.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:38:31.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:38:31.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:38:31.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:38:31.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:38:31.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:38:31.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:38:31.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:38:31.857 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:38:31.857 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:38:31.857 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:38:31.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:38:31.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:38:31.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:38:31.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:38:31.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:38:31.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:38:31.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:38:31.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:38:31.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:38:31.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:38:31.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:38:31.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:38:31.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:38:31.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:38:31.859 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:38:31.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6c1b473e-da2e-401d-8572-88978bcacb57 2024-09-18T11:38:31.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6c1b473e-da2e-401d-8572-88978bcacb57' 2024-09-18T11:38:31.859 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 6c1b473e-da2e-401d-8572-88978bcacb57 2024-09-18T11:38:31.860 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:38:31.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC3u+pmrKDjMxAAzYqmmVoADbe/NBTpNoZbLw== 2024-09-18T11:38:31.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC3u+pmrKDjMxAAzYqmmVoADbe/NBTpNoZbLw=="}' 2024-09-18T11:38:31.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6c1b473e-da2e-401d-8572-88978bcacb57 -i td/ceph-helpers/0/new.json 2024-09-18T11:38:32.189 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:38:32.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:38:32.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3u+pmrKDjMxAAzYqmmVoADbe/NBTpNoZbLw== --osd-uuid 6c1b473e-da2e-401d-8572-88978bcacb57 2024-09-18T11:38:32.223 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:38:32.222+0000 7f2bc0440440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:38:32.234 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:38:32.233+0000 7f2bc0440440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:38:32.235 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:38:32.234+0000 7f2bc0440440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:38:32.236 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:38:32.235+0000 7f2bc0440440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:38:32.236 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:38:32.235+0000 7f2bc0440440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:38:32.236 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:38:32.235+0000 7f2bc0440440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:38:32.275 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:38:32.274+0000 7f2bc0440440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:38:37.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:38:37.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:38:37.657 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:38:37.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:38:37.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:38:38.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:38:38.141 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:38:38.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:38:38.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:38:38.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:38:38.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:38:38.164 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:38:38.162+0000 7f27daa1b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:38:38.165 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:38:38.164+0000 7f27daa1b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:38:38.166 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:38:38.165+0000 7f27daa1b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:38:38.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:38:38.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:38:38.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:38:38.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:38:38.495 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:38:38.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:38:38.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:38.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:38:38.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:38.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:38.687 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:38:38.686+0000 7f27daa1b440 -1 Falling back to public interface 2024-09-18T11:38:38.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:39.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:39.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:39.848 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:38:39.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:38:39.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:39.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:40.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:41.161 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:38:41.160+0000 7f27daa1b440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:38:41.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:41.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:41.206 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:38:41.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:38:41.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:41.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:41.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:42.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:42.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:42.745 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:38:42.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:38:42.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:42.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:43.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:44.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:44.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:44.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:38:44.150 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:38:44.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:44.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:44.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:45.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:45.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:45.508 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:38:45.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:38:45.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:45.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:46.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:47.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:47.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:47.011 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:38:47.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:38:47.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:47.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:47.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:48.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:48.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:48.387 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:38:48.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:38:48.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:48.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:48.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:49.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:49.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:49.899 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:38:49.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:38:49.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:49.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:50.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:51.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:51.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:51.321 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:38:51.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:38:51.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:51.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:51.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:52.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:52.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:52.848 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:38:52.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:38:52.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:52.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:53.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:54.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:54.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:54.388 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:38:54.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:38:54.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:54.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:54.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:55.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:55.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:55.843 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:38:55.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:38:55.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:55.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:56.583 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:57.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:57.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:57.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:38:57.585 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:38:57.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:57.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:58.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:59.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:59.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:59.363 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:38:59.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:38:59.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:59.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:38:59.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:00.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:00.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:00.734 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:39:00.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:39:00.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:00.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:39:01.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:02.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:02.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:02.526 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:39:02.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:39:02.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:02.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:39:03.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:04.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:04.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:04.287 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:39:04.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:39:04.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:04.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:39:04.649 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/95027,v1:127.0.0.1:6803/95027] [v2:127.0.0.1:6804/95027,v1:127.0.0.1:6805/95027] exists,up 6c1b473e-da2e-401d-8572-88978bcacb57 2024-09-18T11:39:04.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:39:04.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:39:04.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:39:04.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: test_get_not_primary: run_osd td/ceph-helpers 1 2024-09-18T11:39:04.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:39:04.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:39:04.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:39:04.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:39:04.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-18T11:39:04.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:39:04.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:39:04.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:39:04.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:39:04.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-18T11:39:04.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-18T11:39:04.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:39:04.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:39:04.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:39:04.655 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:39:04.655 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:39:04.655 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:39:04.655 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:39:04.655 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:39:04.655 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:39:04.655 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:39:04.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:39:04.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:39:04.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:39:04.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:39:04.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:39:04.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:39:04.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:39:04.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:39:04.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:39:04.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:39:04.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:39:04.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:39:04.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-18T11:39:04.657 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:39:04.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ea114169-7d38-414a-a466-1d3c9bc3699b 2024-09-18T11:39:04.657 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 ea114169-7d38-414a-a466-1d3c9bc3699b 2024-09-18T11:39:04.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ea114169-7d38-414a-a466-1d3c9bc3699b' 2024-09-18T11:39:04.657 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:39:04.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDYu+pmfBq/JxAAkwEWG95saGVXq6y4lg0d2g== 2024-09-18T11:39:04.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDYu+pmfBq/JxAAkwEWG95saGVXq6y4lg0d2g=="}' 2024-09-18T11:39:04.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ea114169-7d38-414a-a466-1d3c9bc3699b -i td/ceph-helpers/1/new.json 2024-09-18T11:39:05.140 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:39:05.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-18T11:39:05.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDYu+pmfBq/JxAAkwEWG95saGVXq6y4lg0d2g== --osd-uuid ea114169-7d38-414a-a466-1d3c9bc3699b 2024-09-18T11:39:05.176 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:05.175+0000 7fe01a723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:05.178 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:05.177+0000 7fe01a723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:05.179 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:05.178+0000 7fe01a723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:05.179 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:05.178+0000 7fe01a723440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:39:05.180 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:05.178+0000 7fe01a723440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:39:05.180 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:05.178+0000 7fe01a723440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:39:05.253 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:05.252+0000 7fe01a723440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-18T11:39:11.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-18T11:39:11.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:39:11.212 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T11:39:11.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:39:11.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:39:11.699 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T11:39:11.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:39:11.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:39:11.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:39:11.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:39:11.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:39:11.722 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:11.721+0000 7fd2c383f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:11.728 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:11.727+0000 7fd2c383f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:11.729 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:11.728+0000 7fd2c383f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:12.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:39:12.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:39:12.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:39:12.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:39:12.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:39:12.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:12.115 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:39:12.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:39:12.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:12.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:12.258 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:12.257+0000 7fd2c383f440 -1 Falling back to public interface 2024-09-18T11:39:12.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:13.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:13.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:13.469 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:39:13.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:39:13.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:13.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:13.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:14.792 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:14.789+0000 7fd2c383f440 -1 osd.1 0 log_to_monitors true 2024-09-18T11:39:14.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:14.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:14.872 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:39:14.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:39:14.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:14.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:15.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:16.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:16.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:16.479 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:39:16.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:39:16.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:16.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:16.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:17.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:17.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:17.893 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:39:17.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:39:17.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:17.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:18.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:19.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:19.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:19.301 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:39:19.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:39:19.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:19.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:19.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:20.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:20.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:20.812 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:39:20.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:39:20.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:20.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:21.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:22.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:22.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:22.209 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:39:22.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:39:22.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:22.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:22.614 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:23.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:23.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:23.616 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:39:23.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:39:23.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:23.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:23.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:24.985 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:39:24.985 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:24.985 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:24.985 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:39:24.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:24.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:25.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:26.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:26.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:26.393 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:39:26.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:39:26.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:26.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:26.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:27.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:27.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:27.774 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:39:27.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:39:27.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:27.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:28.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:29.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:29.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:29.542 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:39:29.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:39:29.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:29.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:30.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:31.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:31.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:31.326 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:39:31.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:39:31.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:31.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:32.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:33.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:33.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:33.577 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:39:33.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:39:33.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:33.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:34.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:35.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:35.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:35.352 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:39:35.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:39:35.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:35.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:36.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:37.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:37.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:37.111 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:39:37.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:39:37.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:37.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:37.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:38.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:38.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:38.658 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:39:38.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:39:38.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:38.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:39.010 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/96781,v1:127.0.0.1:6811/96781] [v2:127.0.0.1:6812/96781,v1:127.0.0.1:6813/96781] exists,up ea114169-7d38-414a-a466-1d3c9bc3699b 2024-09-18T11:39:39.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:39:39.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:39:39.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:39:39.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: test_get_not_primary: create_rbd_pool 2024-09-18T11:39:39.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:39:39.361 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:39:39.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:39:39.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:39:39.901 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:39:39.912 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:39:40.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:39:42.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1246: test_get_not_primary: wait_for_clean 2024-09-18T11:39:42.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:39:42.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:39:42.614 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:39:42.614 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:39:42.614 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:39:42.614 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:39:42.615 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:39:42.615 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:39:42.615 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:39:42.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:39:42.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:39:42.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:39:42.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:39:42.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:39:42.709 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:39:43.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:39:43.061 INFO:tasks.workunit.client.0.smithi157.stderr:1' 2024-09-18T11:39:43.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:39:43.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:39:43.062 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:39:43.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836492 2024-09-18T11:39:43.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836492 2024-09-18T11:39:43.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492' 2024-09-18T11:39:43.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:39:43.170 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:39:43.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672965 2024-09-18T11:39:43.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672965 2024-09-18T11:39:43.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492 1-42949672965' 2024-09-18T11:39:43.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:39:43.313 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836492 2024-09-18T11:39:43.313 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:39:43.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:39:43.314 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836492 2024-09-18T11:39:43.314 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:39:43.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836492 2024-09-18T11:39:43.315 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836492 2024-09-18T11:39:43.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-18T11:39:43.316 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:39:43.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-18T11:39:43.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:39:43.670 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672965 2024-09-18T11:39:43.670 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:39:43.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:39:43.671 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672965 2024-09-18T11:39:43.672 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:39:43.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672965 2024-09-18T11:39:43.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-18T11:39:43.673 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672965 2024-09-18T11:39:43.673 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:39:44.026 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-18T11:39:44.026 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:39:44.026 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:39:44.026 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:39:44.499 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:39:44.499 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:39:44.499 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:39:44.499 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:39:44.499 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:39:44.499 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:39:44.499 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:39:44.499 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:39:44.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:39:44.842 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:39:44.842 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:39:44.842 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:39:45.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T11:39:45.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:39:45.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:39:45.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: get_primary rbd GROUP 2024-09-18T11:39:45.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-09-18T11:39:45.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-09-18T11:39:45.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-09-18T11:39:45.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T11:39:45.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: local primary=1 2024-09-18T11:39:45.702 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: get_not_primary rbd GROUP 2024-09-18T11:39:45.702 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-09-18T11:39:45.702 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=GROUP 2024-09-18T11:39:45.702 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd GROUP 2024-09-18T11:39:45.702 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-09-18T11:39:45.702 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-09-18T11:39:45.702 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-09-18T11:39:45.702 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T11:39:46.090 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T11:39:46.090 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd GROUP 2024-09-18T11:39:46.090 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T11:39:46.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: local not_primary=0 2024-09-18T11:39:46.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: test_get_not_primary: test 0 '!=' 1 2024-09-18T11:39:46.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: test_get_not_primary: test 0 = 0 -o 0 = 1 2024-09-18T11:39:46.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: test_get_not_primary: teardown td/ceph-helpers 2024-09-18T11:39:46.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:39:46.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:39:46.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:39:46.476 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:39:46.476 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:39:46.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:39:46.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:39:46.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:39:46.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:39:46.582 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:39:46.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:39:46.583 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:39:46.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:39:46.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:39:46.584 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:39:46.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:39:46.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:39:46.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:39:46.585 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:39:46.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:39:46.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:39:46.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:39:46.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:39:46.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:39:46.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:39:46.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:39:46.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:39:46.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:39:46.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:39:46.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_num_active_clean td/ceph-helpers 2024-09-18T11:39:46.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: local dir=td/ceph-helpers 2024-09-18T11:39:46.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1374: test_get_num_active_clean: setup td/ceph-helpers 2024-09-18T11:39:46.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:39:46.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:39:46.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:39:46.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:39:46.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:39:46.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:39:46.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:39:46.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:39:46.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:39:46.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:39:46.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:39:46.599 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:39:46.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:39:46.600 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:39:46.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:39:46.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:39:46.601 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:39:46.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:39:46.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:39:46.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:39:46.602 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:39:46.603 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:39:46.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:39:46.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:39:46.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:39:46.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:39:46.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:39:46.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:39:46.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:39:46.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:39:46.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:39:46.607 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:39:46.607 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:39:46.607 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:39:46.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:39:46.609 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:39:46.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:39:46.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:39:46.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:39:46.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1375: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:39:46.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:39:46.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:39:46.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:39:46.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:39:46.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:39:46.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:39:46.952 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:39:46.952 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:39:46.952 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:39:46.952 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:39:46.952 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:39:46.952 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:39:46.952 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:39:46.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:39:47.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:39:47.195 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:39:47.195 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:39:47.195 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:39:47.196 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:39:47.196 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:39:47.196 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:39:47.196 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:39:47.196 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:39:47.196 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:39:47.196 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:39:47.196 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:39:47.196 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:39:47.197 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:39:47.197 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:39:47.269 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:39:47.269 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:39:47.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:39:47.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:39:47.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:39:47.270 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:39:47.270 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:39:47.270 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:39:47.270 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:39:47.270 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:39:47.270 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:39:47.270 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:39:47.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:39:47.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:39:47.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1376: test_get_num_active_clean: run_mgr td/ceph-helpers x 2024-09-18T11:39:47.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:39:47.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:39:47.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:39:47.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:39:47.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:39:47.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:39:48.181 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:39:48.181 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:39:48.181 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:39:48.181 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:39:48.181 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:39:48.181 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:39:48.182 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:39:48.182 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:39:48.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:39:48.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1377: test_get_num_active_clean: run_osd td/ceph-helpers 0 2024-09-18T11:39:48.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:39:48.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:39:48.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:39:48.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:39:48.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:39:48.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:39:48.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:39:48.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:39:48.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:39:48.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:39:48.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:39:48.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:39:48.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:39:48.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:39:48.209 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:39:48.209 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:39:48.209 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:39:48.209 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:39:48.210 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:39:48.210 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:39:48.210 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:39:48.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:39:48.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:39:48.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:39:48.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:39:48.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:39:48.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:39:48.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:39:48.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:39:48.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:39:48.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:39:48.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:39:48.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:39:48.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:39:48.211 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:39:48.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f1b70aca-e9c5-48a2-9134-1ab9ae7058b4 2024-09-18T11:39:48.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f1b70aca-e9c5-48a2-9134-1ab9ae7058b4' 2024-09-18T11:39:48.212 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 f1b70aca-e9c5-48a2-9134-1ab9ae7058b4 2024-09-18T11:39:48.212 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:39:48.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAEvOpmEQw/DRAAwTIQJ//rbgLvZ0RitW6K5w== 2024-09-18T11:39:48.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAEvOpmEQw/DRAAwTIQJ//rbgLvZ0RitW6K5w=="}' 2024-09-18T11:39:48.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f1b70aca-e9c5-48a2-9134-1ab9ae7058b4 -i td/ceph-helpers/0/new.json 2024-09-18T11:39:48.526 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:39:48.535 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:39:48.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAEvOpmEQw/DRAAwTIQJ//rbgLvZ0RitW6K5w== --osd-uuid f1b70aca-e9c5-48a2-9134-1ab9ae7058b4 2024-09-18T11:39:48.560 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:48.558+0000 7f65eb82f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:48.572 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:48.570+0000 7f65eb82f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:48.572 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:48.571+0000 7f65eb82f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:48.573 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:48.571+0000 7f65eb82f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:39:48.573 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:48.572+0000 7f65eb82f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:39:48.573 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:48.572+0000 7f65eb82f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:39:48.611 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:48.610+0000 7f65eb82f440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:39:54.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:39:54.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:39:54.175 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:39:54.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:39:54.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:39:54.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:39:54.659 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:39:54.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:39:54.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:39:54.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:39:54.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:39:54.682 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:54.680+0000 7f7e31013440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:54.684 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:54.682+0000 7f7e31013440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:54.685 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:54.683+0000 7f7e31013440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:55.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:39:55.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:39:55.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:39:55.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:39:55.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:39:55.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:55.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:39:55.073 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:39:55.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:55.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:39:55.205 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:55.204+0000 7f7e31013440 -1 Falling back to public interface 2024-09-18T11:39:55.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:56.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:56.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:56.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:39:56.425 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:39:56.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:56.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:39:56.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:57.686 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:39:57.685+0000 7f7e31013440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:39:57.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:57.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:57.786 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:39:57.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:39:57.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:57.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:39:58.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:59.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:59.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:59.317 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:39:59.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:39:59.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:59.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:39:59.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:00.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:00.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:00.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:40:00.704 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:40:00.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:00.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:01.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:02.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:02.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:02.102 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:40:02.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:40:02.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:02.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:02.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:03.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:03.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:03.556 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:40:03.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:40:03.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:03.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:03.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:04.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:04.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:04.991 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:40:04.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:40:04.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:04.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:05.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:06.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:06.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:06.419 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:40:06.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:40:06.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:06.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:06.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:07.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:07.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:07.806 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:40:07.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:40:07.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:07.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:08.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:09.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:09.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:09.194 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:40:09.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:40:09.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:09.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:09.988 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:10.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:10.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:10.990 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:40:10.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:40:10.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:10.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:11.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:12.548 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:12.548 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:12.548 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:40:12.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:40:12.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:12.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:13.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:14.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:14.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:14.275 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:40:14.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:40:14.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:14.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:14.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:15.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:15.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:15.921 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:40:15.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:40:15.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:15.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:16.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:17.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:17.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:17.689 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:40:17.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:40:17.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:17.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:18.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:19.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:19.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:19.164 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:40:19.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:40:19.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:19.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:19.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:20.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:20.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:20.732 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:40:20.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:40:20.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:20.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:21.105 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/99202,v1:127.0.0.1:6803/99202] [v2:127.0.0.1:6804/99202,v1:127.0.0.1:6805/99202] exists,up f1b70aca-e9c5-48a2-9134-1ab9ae7058b4 2024-09-18T11:40:21.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:40:21.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:40:21.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:40:21.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1378: test_get_num_active_clean: create_rbd_pool 2024-09-18T11:40:21.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:40:21.484 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:40:21.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:40:21.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:40:22.020 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:40:22.031 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:40:23.033 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:40:24.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1379: test_get_num_active_clean: wait_for_clean 2024-09-18T11:40:24.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:40:24.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:40:24.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:40:24.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:40:24.045 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:40:24.045 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:40:24.045 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:40:24.045 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:40:24.045 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:40:24.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:40:24.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:40:24.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:40:24.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:40:24.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:40:24.141 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:40:24.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T11:40:24.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:40:24.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:40:24.512 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:40:24.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836486 2024-09-18T11:40:24.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836486 2024-09-18T11:40:24.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836486' 2024-09-18T11:40:24.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:40:24.625 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836486 2024-09-18T11:40:24.625 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:40:24.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:40:24.626 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836486 2024-09-18T11:40:24.626 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:40:24.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836486 2024-09-18T11:40:24.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-18T11:40:24.627 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836486 2024-09-18T11:40:24.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:40:24.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-18T11:40:24.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:40:25.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:40:25.976 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:40:26.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836486 -lt 21474836486 2024-09-18T11:40:26.330 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:40:26.330 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:40:26.330 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:40:26.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:40:26.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:40:26.806 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:40:26.806 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:40:26.806 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:40:26.806 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:40:26.806 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:40:26.807 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:40:27.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:40:27.145 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:40:27.146 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:40:27.146 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:40:27.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T11:40:27.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:40:27.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:40:27.620 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: get_num_active_clean 2024-09-18T11:40:27.620 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:40:27.620 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:40:27.620 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:40:27.620 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:40:27.620 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:40:27.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: local num_active_clean=4 2024-09-18T11:40:27.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1381: test_get_num_active_clean: test 4 = 4 2024-09-18T11:40:27.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1382: test_get_num_active_clean: teardown td/ceph-helpers 2024-09-18T11:40:27.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:40:27.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:40:27.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:40:27.965 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:40:27.965 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:40:27.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:40:27.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:40:27.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:40:28.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:40:28.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:40:28.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:40:28.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:40:28.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:40:28.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:40:28.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:40:28.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:40:28.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:40:28.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:40:28.074 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:40:28.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:40:28.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:40:28.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:40:28.081 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:40:28.081 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:28.081 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:40:28.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:40:28.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:40:28.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:40:28.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:40:28.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_num_active_or_peered td/ceph-helpers 2024-09-18T11:40:28.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: local dir=td/ceph-helpers 2024-09-18T11:40:28.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1404: test_get_num_active_or_peered: setup td/ceph-helpers 2024-09-18T11:40:28.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:40:28.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:40:28.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:40:28.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:40:28.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:40:28.084 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:40:28.084 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:40:28.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:40:28.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:40:28.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:40:28.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:40:28.085 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:40:28.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:40:28.085 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:40:28.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:40:28.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:40:28.087 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:40:28.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:40:28.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:40:28.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:40:28.088 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:40:28.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:40:28.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:40:28.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:40:28.090 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:40:28.090 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:28.090 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:40:28.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:40:28.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:40:28.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:40:28.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:40:28.092 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:40:28.092 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:28.092 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:40:28.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:40:28.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:40:28.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:40:28.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:40:28.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:40:28.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1405: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:40:28.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:40:28.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:40:28.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:40:28.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:40:28.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:40:28.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:40:28.454 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:40:28.454 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:40:28.454 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:40:28.454 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:40:28.454 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:28.454 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:40:28.454 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:40:28.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:40:28.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:40:28.723 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:40:28.724 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:40:28.724 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:40:28.724 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:40:28.724 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:40:28.724 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:40:28.724 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:40:28.724 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:40:28.724 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:40:28.724 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:28.724 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:40:28.724 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:40:28.724 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:40:28.724 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:40:28.800 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:40:28.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:40:28.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:40:28.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:40:28.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:40:28.801 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:40:28.801 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:40:28.801 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:40:28.801 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:40:28.801 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:28.801 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:40:28.801 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:40:28.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:40:28.802 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:40:28.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1406: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2024-09-18T11:40:28.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:40:28.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:40:28.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:40:28.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:40:28.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:40:28.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:40:29.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:40:29.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:40:29.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:40:29.725 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:40:29.725 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:29.725 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:40:29.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:40:29.726 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:40:29.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:40:29.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1407: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2024-09-18T11:40:29.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:40:29.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:40:29.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:40:29.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:40:29.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:40:29.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:40:29.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:40:29.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:40:29.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:40:29.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:40:29.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:40:29.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:40:29.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:40:29.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:40:29.753 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:40:29.753 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:40:29.753 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:40:29.753 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:40:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:40:29.754 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:40:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:40:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:40:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:40:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:40:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:40:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:40:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:40:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:40:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:40:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:40:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:40:29.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:40:29.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:40:29.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:40:29.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9d0f82c9-c9e8-4544-b2a2-438fe35d60c2 2024-09-18T11:40:29.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9d0f82c9-c9e8-4544-b2a2-438fe35d60c2' 2024-09-18T11:40:29.756 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 9d0f82c9-c9e8-4544-b2a2-438fe35d60c2 2024-09-18T11:40:29.756 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:40:29.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAtvOpmJouxLRAARj1FYreTC0zif50q0wIuFg== 2024-09-18T11:40:29.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAtvOpmJouxLRAARj1FYreTC0zif50q0wIuFg=="}' 2024-09-18T11:40:29.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9d0f82c9-c9e8-4544-b2a2-438fe35d60c2 -i td/ceph-helpers/0/new.json 2024-09-18T11:40:30.086 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:40:30.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:40:30.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAtvOpmJouxLRAARj1FYreTC0zif50q0wIuFg== --osd-uuid 9d0f82c9-c9e8-4544-b2a2-438fe35d60c2 2024-09-18T11:40:30.121 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:40:30.119+0000 7fb200c67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:30.132 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:40:30.131+0000 7fb200c67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:30.133 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:40:30.132+0000 7fb200c67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:30.133 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:40:30.132+0000 7fb200c67440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:40:30.134 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:40:30.132+0000 7fb200c67440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:40:30.134 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:40:30.132+0000 7fb200c67440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:40:30.172 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:40:30.170+0000 7fb200c67440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:40:35.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:40:35.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:40:35.734 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:40:35.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:40:35.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:40:36.217 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:40:36.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:40:36.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:40:36.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:40:36.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:40:36.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:40:36.239 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:40:36.238+0000 7fa53d608440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:36.241 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:40:36.240+0000 7fa53d608440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:36.242 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:40:36.241+0000 7fa53d608440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:36.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:40:36.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:40:36.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:40:36.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:40:36.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:40:36.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:36.571 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:40:36.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:40:36.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:36.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:36.767 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:40:36.766+0000 7fa53d608440 -1 Falling back to public interface 2024-09-18T11:40:36.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:37.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:37.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:37.958 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:40:37.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:40:37.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:37.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:38.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:39.271 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:40:39.269+0000 7fa53d608440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:40:39.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:39.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:39.307 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:40:39.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:40:39.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:39.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:39.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:40.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:40.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:40.868 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:40:40.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:40:40.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:40.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:41.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:42.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:42.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:42.244 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:40:42.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:40:42.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:42.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:42.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:43.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:43.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:43.626 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:40:43.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:40:43.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:43.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:44.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:45.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:45.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:45.088 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:40:45.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:40:45.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:45.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:45.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:46.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:46.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:46.447 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:40:46.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:40:46.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:46.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:46.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:47.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:47.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:47.967 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:40:47.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:40:47.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:47.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:48.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:49.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:49.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:49.328 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:40:49.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:40:49.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:49.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:49.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:50.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:50.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:50.717 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:40:50.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:40:50.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:50.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:51.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:52.381 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:52.381 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:52.381 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:40:52.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:40:52.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:52.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:52.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:53.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:53.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:53.975 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:40:53.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:40:53.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:53.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:54.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:55.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:55.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:55.818 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:40:55.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:40:55.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:55.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:56.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:57.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:57.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:57.926 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:40:57.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:40:57.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:57.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:59.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:00.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:00.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:00.024 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:41:00.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:41:00.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:00.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:00.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:01.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:01.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:01.800 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:41:01.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:41:01.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:01.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:02.169 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/101555,v1:127.0.0.1:6803/101555] [v2:127.0.0.1:6804/101555,v1:127.0.0.1:6805/101555] exists,up 9d0f82c9-c9e8-4544-b2a2-438fe35d60c2 2024-09-18T11:41:02.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:41:02.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:41:02.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:41:02.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1408: test_get_num_active_or_peered: create_rbd_pool 2024-09-18T11:41:02.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:41:02.512 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:41:02.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:41:02.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:41:03.143 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:41:03.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:41:04.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:41:05.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1409: test_get_num_active_or_peered: wait_for_clean 2024-09-18T11:41:05.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:41:05.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:41:05.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:41:05.292 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:41:05.293 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:41:05.293 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:41:05.293 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:41:05.293 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:41:05.293 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:41:05.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:41:05.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:41:05.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:41:05.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:41:05.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:41:05.389 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:41:05.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T11:41:05.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:41:05.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:41:05.773 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:41:05.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836486 2024-09-18T11:41:05.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836486 2024-09-18T11:41:05.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836486' 2024-09-18T11:41:05.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:41:05.880 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836486 2024-09-18T11:41:05.880 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:41:05.881 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:41:05.881 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836486 2024-09-18T11:41:05.881 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:41:05.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836486 2024-09-18T11:41:05.883 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836486 2024-09-18T11:41:05.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-18T11:41:05.883 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:41:06.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-18T11:41:06.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:41:07.252 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:41:07.253 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:41:07.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836486 -lt 21474836486 2024-09-18T11:41:07.613 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:41:07.613 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:41:07.613 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:41:08.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:41:08.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:41:08.114 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:41:08.114 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:41:08.115 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:41:08.115 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:41:08.115 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:41:08.115 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:41:08.458 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:41:08.458 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:41:08.459 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:41:08.459 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:41:08.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T11:41:08.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:41:08.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:41:08.926 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: get_num_active_or_peered 2024-09-18T11:41:08.926 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-09-18T11:41:08.926 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-18T11:41:08.926 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-18T11:41:08.926 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-18T11:41:08.926 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-18T11:41:09.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: local num_peered=4 2024-09-18T11:41:09.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1411: test_get_num_active_or_peered: test 4 = 4 2024-09-18T11:41:09.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1412: test_get_num_active_or_peered: teardown td/ceph-helpers 2024-09-18T11:41:09.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:41:09.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:41:09.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:41:09.271 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:41:09.272 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:41:09.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:41:09.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:41:09.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:41:09.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:41:09.377 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:41:09.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:41:09.378 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:41:09.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:41:09.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:41:09.379 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:41:09.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:41:09.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:41:09.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:41:09.380 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:41:09.381 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:41:09.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:41:09.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:41:09.387 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:41:09.387 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:09.388 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:41:09.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:41:09.390 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:41:09.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:41:09.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:41:09.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_num_pgs td/ceph-helpers 2024-09-18T11:41:09.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: local dir=td/ceph-helpers 2024-09-18T11:41:09.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1431: test_get_num_pgs: setup td/ceph-helpers 2024-09-18T11:41:09.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:41:09.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:41:09.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:41:09.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:41:09.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:41:09.392 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:41:09.392 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:41:09.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:41:09.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:41:09.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:41:09.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:41:09.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:41:09.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:41:09.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:41:09.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:41:09.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:41:09.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:41:09.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:41:09.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:41:09.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:41:09.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:41:09.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:41:09.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:41:09.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:41:09.397 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:41:09.397 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:09.397 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:41:09.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:41:09.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:41:09.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:41:09.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:41:09.399 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:41:09.399 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:09.399 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:41:09.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:41:09.401 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:41:09.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:41:09.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:41:09.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:41:09.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:41:09.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:41:09.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:41:09.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:41:09.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:41:09.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:41:09.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:41:09.761 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:41:09.761 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:41:09.761 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:41:09.761 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:41:09.761 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:09.761 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:41:09.761 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:41:09.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:41:10.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:41:10.022 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:41:10.022 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:41:10.022 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:41:10.022 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:41:10.022 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:41:10.022 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:41:10.022 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:41:10.022 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:41:10.023 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:41:10.023 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:10.023 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:41:10.023 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:41:10.023 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:41:10.023 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:41:10.097 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:41:10.097 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:41:10.097 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:41:10.097 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:41:10.097 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:41:10.097 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:41:10.097 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:41:10.097 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:41:10.098 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:41:10.098 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:10.098 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:41:10.098 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:41:10.098 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:41:10.098 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:41:10.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: test_get_num_pgs: run_mgr td/ceph-helpers x 2024-09-18T11:41:10.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:41:10.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:41:10.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:41:10.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:41:10.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:41:10.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:41:11.014 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:41:11.014 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:41:11.014 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:41:11.014 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:41:11.014 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:11.014 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:41:11.014 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:41:11.015 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:41:11.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:41:11.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: test_get_num_pgs: run_osd td/ceph-helpers 0 2024-09-18T11:41:11.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:41:11.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:41:11.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:41:11.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:41:11.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:41:11.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:41:11.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:41:11.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:41:11.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:41:11.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:41:11.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:41:11.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:41:11.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:41:11.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:41:11.042 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:41:11.042 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:41:11.043 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:41:11.043 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:41:11.043 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:11.043 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:41:11.043 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:41:11.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:41:11.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:41:11.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:41:11.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:41:11.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:41:11.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:41:11.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:41:11.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:41:11.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:41:11.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:41:11.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:41:11.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:41:11.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:41:11.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:41:11.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=581fe203-0062-4494-8c5e-2900bb5bb824 2024-09-18T11:41:11.044 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 581fe203-0062-4494-8c5e-2900bb5bb824 2024-09-18T11:41:11.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 581fe203-0062-4494-8c5e-2900bb5bb824' 2024-09-18T11:41:11.045 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:41:11.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBXvOpmmdxSAxAA9a0a4kzGbaCwi6LQaDdJ3Q== 2024-09-18T11:41:11.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBXvOpmmdxSAxAA9a0a4kzGbaCwi6LQaDdJ3Q=="}' 2024-09-18T11:41:11.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 581fe203-0062-4494-8c5e-2900bb5bb824 -i td/ceph-helpers/0/new.json 2024-09-18T11:41:11.359 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:41:11.368 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:41:11.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXvOpmmdxSAxAA9a0a4kzGbaCwi6LQaDdJ3Q== --osd-uuid 581fe203-0062-4494-8c5e-2900bb5bb824 2024-09-18T11:41:11.394 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:11.393+0000 7fed12009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:11.405 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:11.403+0000 7fed12009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:11.406 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:11.404+0000 7fed12009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:11.406 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:11.405+0000 7fed12009440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:41:11.406 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:11.405+0000 7fed12009440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:41:11.406 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:11.405+0000 7fed12009440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:41:11.445 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:11.443+0000 7fed12009440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:41:16.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:41:16.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:41:16.791 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:41:16.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:41:16.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:41:17.271 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:41:17.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:41:17.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:41:17.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:41:17.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:41:17.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:41:17.294 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:17.292+0000 7f393232d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:17.296 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:17.294+0000 7f393232d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:17.297 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:17.295+0000 7f393232d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:17.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:41:17.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:41:17.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:41:17.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:41:17.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:41:17.625 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:41:17.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:17.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:41:17.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:17.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:17.817 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:17.816+0000 7f393232d440 -1 Falling back to public interface 2024-09-18T11:41:17.988 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:18.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:18.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:18.990 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:41:18.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:41:18.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:18.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:19.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:20.302 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:20.300+0000 7f393232d440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:41:20.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:20.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:20.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:41:20.343 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:41:20.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:20.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:20.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:21.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:21.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:21.702 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:41:21.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:41:21.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:21.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:22.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:23.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:23.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:23.199 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:41:23.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:41:23.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:23.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:23.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:24.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:24.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:24.588 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:41:24.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:41:24.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:24.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:25.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:26.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:26.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:26.004 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:41:26.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:41:26.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:26.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:26.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:27.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:27.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:27.399 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:41:27.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:41:27.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:27.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:27.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:28.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:28.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:28.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:41:28.781 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:41:28.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:28.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:29.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:30.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:30.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:30.278 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:41:30.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:41:30.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:30.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:30.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:31.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:31.666 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:41:31.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:31.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:41:31.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:31.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:32.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:33.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:33.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:33.470 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:41:33.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:41:33.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:33.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:34.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:35.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:35.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:35.346 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:41:35.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:41:35.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:35.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:36.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:37.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:37.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:37.038 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:41:37.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:41:37.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:37.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:37.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:38.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:38.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:38.728 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:41:38.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:41:38.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:38.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:39.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:40.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:40.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:40.530 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:41:40.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:41:40.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:40.531 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:41.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:42.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:42.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:42.206 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:41:42.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:41:42.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:42.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:42.690 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/103869,v1:127.0.0.1:6803/103869] [v2:127.0.0.1:6804/103869,v1:127.0.0.1:6805/103869] exists,up 581fe203-0062-4494-8c5e-2900bb5bb824 2024-09-18T11:41:42.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:41:42.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:41:42.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:41:42.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: test_get_num_pgs: create_rbd_pool 2024-09-18T11:41:42.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:41:43.034 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:41:43.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:41:43.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:41:43.515 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:41:43.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:41:44.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:41:45.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1436: test_get_num_pgs: wait_for_clean 2024-09-18T11:41:45.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:41:45.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:41:45.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:41:45.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:41:45.858 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:41:45.858 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:41:45.858 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:41:45.858 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:41:45.858 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:41:45.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:41:45.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:41:45.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:41:45.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:41:45.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:41:45.954 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:41:46.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T11:41:46.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:41:46.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:41:46.312 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:41:46.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836485 2024-09-18T11:41:46.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836485 2024-09-18T11:41:46.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485' 2024-09-18T11:41:46.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:41:46.431 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836485 2024-09-18T11:41:46.431 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:41:46.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:41:46.433 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836485 2024-09-18T11:41:46.433 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:41:46.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836485 2024-09-18T11:41:46.433 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836485 2024-09-18T11:41:46.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-18T11:41:46.434 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:41:46.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836485 2024-09-18T11:41:46.852 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:41:46.852 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:41:46.852 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:41:47.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:41:47.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:41:47.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:41:47.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:41:47.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:41:47.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:41:47.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:41:47.338 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:41:47.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:41:47.681 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:41:47.681 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:41:47.681 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:41:48.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T11:41:48.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:41:48.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:41:48.148 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: get_num_pgs 2024-09-18T11:41:48.148 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:41:48.148 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:41:48.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: local num_pgs=4 2024-09-18T11:41:48.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1438: test_get_num_pgs: test 4 -gt 0 2024-09-18T11:41:48.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1439: test_get_num_pgs: teardown td/ceph-helpers 2024-09-18T11:41:48.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:41:48.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:41:48.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:41:48.630 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:41:48.630 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:41:48.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:41:48.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:41:48.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:41:48.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:41:48.735 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:41:48.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:41:48.736 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:41:48.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:41:48.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:41:48.737 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:41:48.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:41:48.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:41:48.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:41:48.739 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:41:48.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:41:48.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:41:48.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:41:48.746 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:41:48.746 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:48.746 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:41:48.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:41:48.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:41:48.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:41:48.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:41:48.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_op_scheduler td/ceph-helpers 2024-09-18T11:41:48.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: test_get_op_scheduler: local dir=td/ceph-helpers 2024-09-18T11:41:48.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: test_get_op_scheduler: setup td/ceph-helpers 2024-09-18T11:41:48.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:41:48.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:41:48.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:41:48.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:41:48.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:41:48.749 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:41:48.749 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:41:48.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:41:48.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:41:48.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:41:48.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:41:48.749 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:41:48.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:41:48.750 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:41:48.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:41:48.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:41:48.751 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:41:48.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:41:48.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:41:48.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:41:48.753 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:41:48.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:41:48.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:41:48.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:41:48.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:41:48.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:48.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:41:48.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:41:48.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:41:48.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:41:48.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:41:48.757 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:41:48.757 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:48.757 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:41:48.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:41:48.759 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:41:48.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:41:48.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:41:48.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:41:48.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: test_get_op_scheduler: run_mon td/ceph-helpers a 2024-09-18T11:41:48.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:41:48.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:41:48.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:41:48.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:41:48.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:41:48.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-18T11:41:49.108 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:41:49.108 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:41:49.108 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:41:49.108 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:41:49.108 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:49.108 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:41:49.108 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:41:49.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:41:49.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:41:49.402 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:41:49.402 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:41:49.402 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:41:49.402 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:41:49.403 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:41:49.403 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:41:49.403 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:41:49.403 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:41:49.404 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:41:49.404 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:49.404 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:41:49.404 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:41:49.404 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:41:49.404 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:41:49.481 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:41:49.481 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:41:49.481 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:41:49.481 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:41:49.481 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:41:49.481 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:41:49.481 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:41:49.481 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:41:49.481 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:41:49.481 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:49.481 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:41:49.481 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:41:49.482 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:41:49.482 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:41:49.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: test_get_op_scheduler: run_mgr td/ceph-helpers x 2024-09-18T11:41:49.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:41:49.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:41:49.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:41:49.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:41:49.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:41:49.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:41:50.388 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:41:50.388 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:41:50.388 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:41:50.388 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:41:50.388 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:50.388 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:41:50.389 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:41:50.389 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:41:50.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:41:50.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2024-09-18T11:41:50.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:41:50.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:41:50.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:41:50.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:41:50.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:41:50.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:41:50.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:41:50.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:41:50.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:41:50.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:41:50.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:41:50.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:41:50.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:41:50.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:41:50.417 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:41:50.417 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:41:50.417 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:41:50.417 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:41:50.417 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:50.417 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:41:50.417 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:41:50.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:41:50.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:41:50.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:41:50.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:41:50.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:41:50.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:41:50.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:41:50.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:41:50.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:41:50.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:41:50.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:41:50.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=wpq 2024-09-18T11:41:50.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:41:50.418 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:41:50.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=01373bdf-b206-43a9-89b2-4c20c6656b4e 2024-09-18T11:41:50.419 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 01373bdf-b206-43a9-89b2-4c20c6656b4e 2024-09-18T11:41:50.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 01373bdf-b206-43a9-89b2-4c20c6656b4e' 2024-09-18T11:41:50.419 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:41:50.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB+vOpmngaWGRAACU+0+RPOAfuAG6r49QDyuQ== 2024-09-18T11:41:50.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB+vOpmngaWGRAACU+0+RPOAfuAG6r49QDyuQ=="}' 2024-09-18T11:41:50.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 01373bdf-b206-43a9-89b2-4c20c6656b4e -i td/ceph-helpers/0/new.json 2024-09-18T11:41:50.732 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:41:50.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:41:50.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQB+vOpmngaWGRAACU+0+RPOAfuAG6r49QDyuQ== --osd-uuid 01373bdf-b206-43a9-89b2-4c20c6656b4e 2024-09-18T11:41:50.767 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:50.766+0000 7f8703e0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:50.777 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:50.776+0000 7f8703e0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:50.778 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:50.777+0000 7f8703e0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:50.778 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:50.777+0000 7f8703e0b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:41:50.778 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:50.777+0000 7f8703e0b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:41:50.778 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:50.777+0000 7f8703e0b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:41:50.809 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:50.807+0000 7f8703e0b440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:41:56.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:41:56.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:41:56.214 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:41:56.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:41:56.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:41:56.698 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:41:56.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:41:56.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2024-09-18T11:41:56.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:41:56.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:41:56.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:41:56.721 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:56.720+0000 7f1501c2c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:56.723 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:56.722+0000 7f1501c2c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:56.724 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:56.723+0000 7f1501c2c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:57.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:41:57.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:41:57.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:41:57.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:41:57.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:41:57.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:57.051 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:41:57.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:41:57.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:57.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:57.246 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:57.245+0000 7f1501c2c440 -1 Falling back to public interface 2024-09-18T11:41:57.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:58.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:58.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:58.412 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:41:58.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:41:58.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:58.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:41:58.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:59.726 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:41:59.723+0000 7f1501c2c440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:41:59.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:59.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:59.769 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:41:59.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:41:59.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:59.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:42:00.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:01.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:01.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:01.213 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:42:01.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:42:01.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:01.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:42:01.600 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/106158,v1:127.0.0.1:6803/106158] [v2:127.0.0.1:6804/106158,v1:127.0.0.1:6805/106158] exists,up 01373bdf-b206-43a9-89b2-4c20c6656b4e 2024-09-18T11:42:01.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:42:01.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:42:01.603 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:42:01.603 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: test_get_op_scheduler: get_op_scheduler 0 2024-09-18T11:42:01.603 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2225: get_op_scheduler: local id=0 2024-09-18T11:42:01.603 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: get_op_scheduler: get_config osd 0 osd_op_queue 2024-09-18T11:42:01.603 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-09-18T11:42:01.603 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-09-18T11:42:01.603 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-09-18T11:42:01.603 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-09-18T11:42:01.603 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-09-18T11:42:01.603 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:42:01.603 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:42:01.603 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:42:01.604 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:42:01.604 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:42:01.604 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-osd.0.asok 2024-09-18T11:42:01.604 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:42:01.604 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-osd.0.asok config get osd_op_queue 2024-09-18T11:42:01.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: test_get_op_scheduler: test wpq = wpq 2024-09-18T11:42:01.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2024-09-18T11:42:01.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:42:01.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:42:01.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:42:01.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:42:01.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-18T11:42:01.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:42:01.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:42:01.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:42:01.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:42:01.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-18T11:42:01.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-18T11:42:01.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:42:01.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:42:01.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:42:01.685 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:42:01.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:42:01.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:42:01.686 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:42:01.686 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:42:01.686 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:42:01.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:42:01.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:42:01.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:42:01.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:42:01.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:42:01.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:42:01.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:42:01.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:42:01.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:42:01.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:42:01.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:42:01.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:42:01.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-18T11:42:01.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-18T11:42:01.687 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:42:01.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5c248e1d-99c0-4137-8756-fa9265e67f81 2024-09-18T11:42:01.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5c248e1d-99c0-4137-8756-fa9265e67f81' 2024-09-18T11:42:01.688 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 5c248e1d-99c0-4137-8756-fa9265e67f81 2024-09-18T11:42:01.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:42:01.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCJvOpmaDqiKRAAEN7iX9vZTaZyDJUxrWziog== 2024-09-18T11:42:01.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCJvOpmaDqiKRAAEN7iX9vZTaZyDJUxrWziog=="}' 2024-09-18T11:42:01.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5c248e1d-99c0-4137-8756-fa9265e67f81 -i td/ceph-helpers/1/new.json 2024-09-18T11:42:02.204 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:42:02.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-18T11:42:02.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCJvOpmaDqiKRAAEN7iX9vZTaZyDJUxrWziog== --osd-uuid 5c248e1d-99c0-4137-8756-fa9265e67f81 2024-09-18T11:42:02.240 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:02.239+0000 7f9b40d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:02.242 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:02.241+0000 7f9b40d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:02.243 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:02.242+0000 7f9b40d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:02.243 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:02.242+0000 7f9b40d23440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:42:02.244 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:02.242+0000 7f9b40d23440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:42:02.244 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:02.242+0000 7f9b40d23440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:42:02.320 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:02.318+0000 7f9b40d23440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-18T11:42:08.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-18T11:42:08.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:42:08.158 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T11:42:08.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:42:08.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:42:08.693 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T11:42:08.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:42:08.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-18T11:42:08.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:42:08.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:42:08.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:42:08.716 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:08.715+0000 7f000f86f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:08.743 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:08.742+0000 7f000f86f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:08.744 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:08.743+0000 7f000f86f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:09.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:42:09.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:42:09.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:42:09.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:42:09.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:42:09.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:09.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:42:09.048 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:42:09.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:09.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:09.267 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:09.266+0000 7f000f86f440 -1 Falling back to public interface 2024-09-18T11:42:09.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:10.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:10.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:10.407 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:42:10.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:42:10.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:10.408 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:10.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:11.738 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:11.736+0000 7f000f86f440 -1 osd.1 0 log_to_monitors true 2024-09-18T11:42:11.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:11.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:11.774 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:42:11.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:42:11.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:11.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:12.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:13.390 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:13.390 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:13.391 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:42:13.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:42:13.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:13.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:13.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:14.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:14.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:14.792 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:42:14.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:42:14.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:14.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:15.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:16.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:16.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:16.234 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:42:16.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:42:16.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:16.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:16.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:17.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:17.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:17.625 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:42:17.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:42:17.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:17.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:18.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:19.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:19.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:19.011 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:42:19.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:42:19.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:19.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:19.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:20.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:20.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:20.476 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:42:20.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:42:20.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:20.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:20.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:21.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:21.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:21.942 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:42:21.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:42:21.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:21.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:22.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:23.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:23.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:23.301 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:42:23.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:42:23.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:23.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:23.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:24.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:24.800 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:42:24.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:24.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:42:24.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:24.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:25.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:26.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:26.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:26.496 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:42:26.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:42:26.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:26.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:27.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:28.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:28.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:28.299 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:42:28.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:42:28.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:28.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:29.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:30.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:30.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:30.026 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:42:30.026 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:42:30.026 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:30.026 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:30.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:31.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:31.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:31.911 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:42:31.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:42:31.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:31.912 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:32.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:33.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:33.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:33.469 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:42:33.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:42:33.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:33.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:34.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:35.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:35.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:35.623 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:42:35.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:42:35.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:35.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:42:36.013 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/107562,v1:127.0.0.1:6811/107562] [v2:127.0.0.1:6812/107562,v1:127.0.0.1:6813/107562] exists,up 5c248e1d-99c0-4137-8756-fa9265e67f81 2024-09-18T11:42:36.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:42:36.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:42:36.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:42:36.016 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: test_get_op_scheduler: get_op_scheduler 1 2024-09-18T11:42:36.016 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2225: get_op_scheduler: local id=1 2024-09-18T11:42:36.016 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: get_op_scheduler: get_config osd 1 osd_op_queue 2024-09-18T11:42:36.016 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-09-18T11:42:36.016 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=1 2024-09-18T11:42:36.016 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-09-18T11:42:36.016 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-09-18T11:42:36.016 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.1 2024-09-18T11:42:36.016 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T11:42:36.017 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T11:42:36.017 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:42:36.017 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:42:36.017 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:42:36.017 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-osd.1.asok 2024-09-18T11:42:36.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:42:36.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-osd.1.asok config get osd_op_queue 2024-09-18T11:42:36.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2024-09-18T11:42:36.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: test_get_op_scheduler: teardown td/ceph-helpers 2024-09-18T11:42:36.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:42:36.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:42:36.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:42:36.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:42:36.096 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:42:36.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:42:36.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:42:36.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:42:36.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:42:36.201 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:42:36.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:42:36.202 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:42:36.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:42:36.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:42:36.203 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:42:36.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:42:36.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:42:36.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:42:36.205 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:42:36.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:42:36.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:42:36.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:42:36.212 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:42:36.213 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:42:36.213 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:42:36.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:42:36.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:42:36.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:42:36.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:42:36.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2024-09-18T11:42:36.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-09-18T11:42:36.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1461: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2024-09-18T11:42:36.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:42:36.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:42:36.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:42:36.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:42:36.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:42:36.216 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:42:36.216 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:42:36.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:42:36.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:42:36.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:42:36.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:42:36.217 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:42:36.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:42:36.217 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:42:36.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:42:36.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:42:36.219 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:42:36.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:42:36.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:42:36.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:42:36.220 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:42:36.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:42:36.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:42:36.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:42:36.222 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:42:36.222 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:42:36.222 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:42:36.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:42:36.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:42:36.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:42:36.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:42:36.224 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:42:36.224 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:42:36.224 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:42:36.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:42:36.226 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:42:36.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:42:36.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:42:36.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:42:36.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1462: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:42:36.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:42:36.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:42:36.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:42:36.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:42:36.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:42:36.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:42:36.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:42:36.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:42:36.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:42:36.576 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:42:36.576 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:42:36.577 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:42:36.577 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:42:36.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:42:36.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:42:36.837 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:42:36.837 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:42:36.837 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:42:36.838 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:42:36.838 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:42:36.838 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:42:36.838 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:42:36.838 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:42:36.838 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:42:36.838 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:42:36.838 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:42:36.838 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:42:36.838 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:42:36.838 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:42:36.912 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:42:36.981 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:42:36.981 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:42:36.981 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:42:36.981 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:42:36.981 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:42:36.981 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:42:36.981 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:42:36.981 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:42:36.981 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:42:36.981 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:42:36.981 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:42:36.981 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:42:36.982 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:42:36.988 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1463: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-09-18T11:42:36.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:42:36.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:42:36.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:42:36.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:42:36.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:42:36.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:42:37.823 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:42:37.823 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:42:37.823 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:42:37.823 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:42:37.823 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:42:37.823 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:42:37.823 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:42:37.824 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:42:37.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:42:37.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1464: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-09-18T11:42:37.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:42:37.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:42:37.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:42:37.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:42:37.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:42:37.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:42:37.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:42:37.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:42:37.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:42:37.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:42:37.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:42:37.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:42:37.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:42:37.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:42:37.852 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:42:37.852 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:42:37.852 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:42:37.852 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:42:37.852 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:42:37.852 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:42:37.852 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:42:37.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:42:37.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:42:37.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:42:37.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:42:37.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:42:37.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:42:37.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:42:37.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:42:37.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:42:37.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:42:37.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:42:37.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:42:37.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:42:37.853 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:42:37.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9fdca273-ab7d-4228-8c0d-1e5d6c82292f 2024-09-18T11:42:37.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9fdca273-ab7d-4228-8c0d-1e5d6c82292f' 2024-09-18T11:42:37.854 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 9fdca273-ab7d-4228-8c0d-1e5d6c82292f 2024-09-18T11:42:37.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:42:37.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCtvOpmOuqMMxAAKjPPjJuTCbY29OV4Y8ZyWA== 2024-09-18T11:42:37.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCtvOpmOuqMMxAAKjPPjJuTCbY29OV4Y8ZyWA=="}' 2024-09-18T11:42:37.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9fdca273-ab7d-4228-8c0d-1e5d6c82292f -i td/ceph-helpers/0/new.json 2024-09-18T11:42:38.166 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:42:38.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:42:38.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtvOpmOuqMMxAAKjPPjJuTCbY29OV4Y8ZyWA== --osd-uuid 9fdca273-ab7d-4228-8c0d-1e5d6c82292f 2024-09-18T11:42:38.201 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:38.200+0000 7f6cdf809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:38.212 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:38.211+0000 7f6cdf809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:38.214 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:38.212+0000 7f6cdf809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:38.214 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:38.212+0000 7f6cdf809440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:42:38.214 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:38.212+0000 7f6cdf809440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:42:38.214 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:38.212+0000 7f6cdf809440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:42:38.252 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:38.251+0000 7f6cdf809440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:42:43.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:42:43.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:42:43.623 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:42:43.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:42:43.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:42:44.119 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:42:44.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:42:44.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:42:44.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:42:44.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:42:44.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:42:44.142 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:44.140+0000 7f488692d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:44.144 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:44.142+0000 7f488692d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:44.145 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:44.143+0000 7f488692d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:44.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:42:44.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:42:44.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:42:44.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:42:44.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:42:44.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:44.521 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:42:44.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:42:44.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:44.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:42:44.662 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:44.661+0000 7f488692d440 -1 Falling back to public interface 2024-09-18T11:42:44.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:45.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:45.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:45.875 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:42:45.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:42:45.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:45.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:42:46.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:47.135 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:42:47.134+0000 7f488692d440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:42:47.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:47.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:47.232 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:42:47.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:42:47.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:47.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:42:47.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:48.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:48.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:48.695 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:42:48.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:42:48.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:48.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:42:49.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:50.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:50.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:50.144 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:42:50.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:42:50.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:50.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:42:50.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:51.548 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:51.548 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:51.549 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:42:51.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:42:51.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:51.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:42:51.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:52.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:52.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:52.909 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:42:52.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:42:52.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:52.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:42:53.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:54.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:54.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:54.285 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:42:54.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:42:54.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:54.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:42:54.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:55.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:55.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:55.651 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:42:55.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:42:55.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:55.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:42:56.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:57.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:57.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:57.129 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:42:57.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:42:57.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:57.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:42:57.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:58.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:58.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:58.510 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:42:58.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:42:58.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:58.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:42:59.463 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:00.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:00.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:00.465 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:43:00.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:43:00.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:00.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:01.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:02.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:02.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:02.022 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:43:02.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:43:02.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:02.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:02.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:03.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:03.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:03.645 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:43:03.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:43:03.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:03.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:04.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:05.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:05.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:05.234 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:43:05.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:43:05.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:05.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:06.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:07.365 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:07.365 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:07.365 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:43:07.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:43:07.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:07.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:07.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:08.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:08.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:08.907 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:43:08.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:43:08.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:08.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:09.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:10.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:10.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:10.388 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:43:10.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:43:10.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:10.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:10.742 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/109593,v1:127.0.0.1:6803/109593] [v2:127.0.0.1:6804/109593,v1:127.0.0.1:6805/109593] exists,up 9fdca273-ab7d-4228-8c0d-1e5d6c82292f 2024-09-18T11:43:10.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:43:10.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:43:10.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:43:10.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1465: test_get_osd_id_used_by_pgs: create_rbd_pool 2024-09-18T11:43:10.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:43:11.098 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:43:11.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:43:11.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:43:11.715 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:43:11.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:43:12.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:43:13.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1466: test_get_osd_id_used_by_pgs: wait_for_clean 2024-09-18T11:43:13.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:43:13.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:43:13.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:43:13.804 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:43:13.804 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:43:13.804 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:43:13.804 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:43:13.805 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:43:13.805 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:43:13.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:43:13.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:43:13.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:43:13.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:43:13.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:43:13.900 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:43:14.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T11:43:14.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:43:14.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:43:14.249 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:43:14.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836486 2024-09-18T11:43:14.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836486 2024-09-18T11:43:14.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836486' 2024-09-18T11:43:14.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:43:14.357 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836486 2024-09-18T11:43:14.357 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:43:14.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:43:14.358 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836486 2024-09-18T11:43:14.359 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:43:14.359 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836486 2024-09-18T11:43:14.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836486 2024-09-18T11:43:14.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-18T11:43:14.360 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:43:14.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 4 -lt 21474836486 2024-09-18T11:43:14.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:43:15.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:43:15.728 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:43:16.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836487 -lt 21474836486 2024-09-18T11:43:16.113 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:43:16.113 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:43:16.113 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:43:16.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:43:16.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:43:16.589 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:43:16.590 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:43:16.590 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:43:16.590 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:43:16.590 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:43:16.590 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:43:16.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:43:16.932 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:43:16.932 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:43:16.932 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:43:17.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T11:43:17.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:43:17.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:43:17.461 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-09-18T11:43:17.461 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: uniq 2024-09-18T11:43:17.461 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-09-18T11:43:17.462 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-09-18T11:43:17.462 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2024-09-18T11:43:17.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: local osd_ids=0 2024-09-18T11:43:17.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: test_get_osd_id_used_by_pgs: test 0 = 0 2024-09-18T11:43:17.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-09-18T11:43:17.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:43:17.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:43:17.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:43:17.836 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:43:17.836 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:43:17.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:43:17.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:43:17.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:43:17.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:43:17.941 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:43:17.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:43:17.942 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:43:17.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:43:17.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:43:17.943 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:43:17.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:43:17.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:43:17.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:43:17.944 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:43:17.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:43:17.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:43:17.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:43:17.952 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:43:17.952 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:17.952 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:43:17.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:43:17.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:43:17.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:43:17.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:43:17.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_osds td/ceph-helpers 2024-09-18T11:43:17.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: local dir=td/ceph-helpers 2024-09-18T11:43:17.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: setup td/ceph-helpers 2024-09-18T11:43:17.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:43:17.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:43:17.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:43:17.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:43:17.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:43:17.955 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:43:17.955 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:43:17.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:43:17.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:43:17.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:43:17.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:43:17.955 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:43:17.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:43:17.956 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:43:17.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:43:17.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:43:17.957 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:43:17.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:43:17.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:43:17.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:43:17.959 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:43:17.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:43:17.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:43:17.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:43:17.961 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:43:17.961 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:17.961 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:43:17.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:43:17.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:43:17.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:43:17.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:43:17.963 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:43:17.963 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:17.964 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:43:17.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:43:17.965 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:43:17.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:43:17.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:43:17.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:43:17.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-18T11:43:17.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:43:17.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:43:17.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:43:17.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:43:17.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:43:17.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-18T11:43:18.340 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:43:18.340 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:43:18.340 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:43:18.341 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:43:18.341 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:18.341 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:43:18.341 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:43:18.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-18T11:43:18.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:43:18.601 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:43:18.601 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:43:18.601 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:43:18.601 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:43:18.601 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:43:18.602 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:43:18.602 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:43:18.602 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:43:18.602 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:43:18.602 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:18.602 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:43:18.602 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:43:18.602 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:43:18.602 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:43:18.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:43:18.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:43:18.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:43:18.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:43:18.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:43:18.676 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:43:18.676 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:43:18.676 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:43:18.677 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:43:18.677 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:18.677 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:43:18.677 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:43:18.677 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:43:18.677 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:43:18.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: test_get_osds: run_mgr td/ceph-helpers x 2024-09-18T11:43:18.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:43:18.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:43:18.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:43:18.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:43:18.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:43:18.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:43:19.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:43:19.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:43:19.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:43:19.598 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:43:19.598 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:19.598 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:43:19.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:43:19.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:43:19.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:43:19.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: test_get_osds: run_osd td/ceph-helpers 0 2024-09-18T11:43:19.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:43:19.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:43:19.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:43:19.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:43:19.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:43:19.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:43:19.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:43:19.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:43:19.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:43:19.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:43:19.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:43:19.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:43:19.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:43:19.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:43:19.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:43:19.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:43:19.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:43:19.627 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:43:19.627 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:19.627 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:43:19.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:43:19.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:43:19.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:43:19.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:43:19.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:43:19.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:43:19.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:43:19.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:43:19.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:43:19.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:43:19.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:43:19.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:43:19.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:43:19.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:43:19.628 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:43:19.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a3c9f648-3083-4473-a635-b9ebd073988e 2024-09-18T11:43:19.629 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 a3c9f648-3083-4473-a635-b9ebd073988e 2024-09-18T11:43:19.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a3c9f648-3083-4473-a635-b9ebd073988e' 2024-09-18T11:43:19.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:43:19.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDXvOpmaXsPJhAAI+si76UVlz0oEfUSbR/Ung== 2024-09-18T11:43:19.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDXvOpmaXsPJhAAI+si76UVlz0oEfUSbR/Ung=="}' 2024-09-18T11:43:19.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a3c9f648-3083-4473-a635-b9ebd073988e -i td/ceph-helpers/0/new.json 2024-09-18T11:43:19.939 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:43:19.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:43:19.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXvOpmaXsPJhAAI+si76UVlz0oEfUSbR/Ung== --osd-uuid a3c9f648-3083-4473-a635-b9ebd073988e 2024-09-18T11:43:19.974 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:19.973+0000 7f7a1462f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:19.985 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:19.984+0000 7f7a1462f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:19.986 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:19.985+0000 7f7a1462f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:19.986 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:19.985+0000 7f7a1462f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:43:19.986 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:19.985+0000 7f7a1462f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:43:19.986 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:19.985+0000 7f7a1462f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:43:20.025 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:20.024+0000 7f7a1462f440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:43:25.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:43:25.648 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:43:25.649 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:43:25.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:43:25.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:43:26.136 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:43:26.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:43:26.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:43:26.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:43:26.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:43:26.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:43:26.159 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:26.157+0000 7fb3f511f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:26.160 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:26.159+0000 7fb3f511f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:26.161 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:26.160+0000 7fb3f511f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:26.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:43:26.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:43:26.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:43:26.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:43:26.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:43:26.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:26.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:43:26.486 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:43:26.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:26.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:26.677 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:26.676+0000 7fb3f511f440 -1 Falling back to public interface 2024-09-18T11:43:26.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:27.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:27.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:27.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:43:27.839 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:43:27.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:27.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:28.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:29.160 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:29.158+0000 7fb3f511f440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:43:29.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:29.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:29.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:43:29.195 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:43:29.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:29.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:29.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:30.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:30.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:30.747 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:43:30.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:43:30.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:30.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:31.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:32.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:32.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:32.199 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:43:32.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:43:32.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:32.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:32.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:33.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:33.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:33.565 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:43:33.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:43:33.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:33.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:33.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:34.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:34.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:34.944 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:43:34.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:43:34.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:34.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:35.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:36.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:36.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:36.310 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:43:36.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:43:36.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:36.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:36.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:37.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:37.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:37.669 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:43:37.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:43:37.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:37.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:38.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:39.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:39.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:39.161 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:43:39.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:43:39.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:39.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:39.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:40.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:40.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:40.526 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:43:40.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:43:40.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:40.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:41.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:42.463 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:42.463 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:42.463 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:43:42.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:43:42.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:42.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:43.026 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:44.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:44.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:44.028 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:43:44.028 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:43:44.028 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:44.028 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:44.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:45.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:45.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:45.734 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:43:45.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:43:45.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:45.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:46.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:47.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:47.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:47.545 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:43:47.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:43:47.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:47.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:48.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:49.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:49.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:49.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:43:49.324 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:43:49.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:49.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:49.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:50.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:50.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:50.732 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:43:50.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:43:50.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:50.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:51.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:52.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:52.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:52.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:43:52.211 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:43:52.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:52.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:52.584 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/111934,v1:127.0.0.1:6803/111934] [v2:127.0.0.1:6804/111934,v1:127.0.0.1:6805/111934] exists,up a3c9f648-3083-4473-a635-b9ebd073988e 2024-09-18T11:43:52.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:43:52.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:43:52.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:43:52.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1037: test_get_osds: run_osd td/ceph-helpers 1 2024-09-18T11:43:52.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:43:52.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:43:52.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:43:52.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:43:52.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-18T11:43:52.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:43:52.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:43:52.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:43:52.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:43:52.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-18T11:43:52.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-18T11:43:52.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:43:52.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:43:52.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:43:52.589 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:43:52.589 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:43:52.589 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:43:52.589 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:43:52.589 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:52.589 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:43:52.590 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:43:52.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:43:52.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:43:52.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:43:52.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:43:52.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:43:52.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:43:52.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:43:52.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:43:52.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:43:52.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:43:52.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:43:52.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:43:52.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-18T11:43:52.591 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:43:52.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cf4898a4-6ad1-4687-891b-571e82dd3d2f 2024-09-18T11:43:52.591 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 cf4898a4-6ad1-4687-891b-571e82dd3d2f 2024-09-18T11:43:52.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 cf4898a4-6ad1-4687-891b-571e82dd3d2f' 2024-09-18T11:43:52.591 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:43:52.603 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD4vOpmNJDhIxAAYurZeXNLtxemYJvPyjs+CQ== 2024-09-18T11:43:52.603 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD4vOpmNJDhIxAAYurZeXNLtxemYJvPyjs+CQ=="}' 2024-09-18T11:43:52.603 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cf4898a4-6ad1-4687-891b-571e82dd3d2f -i td/ceph-helpers/1/new.json 2024-09-18T11:43:53.073 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:43:53.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-18T11:43:53.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD4vOpmNJDhIxAAYurZeXNLtxemYJvPyjs+CQ== --osd-uuid cf4898a4-6ad1-4687-891b-571e82dd3d2f 2024-09-18T11:43:53.109 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:53.108+0000 7fa9dbf1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:53.111 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:53.110+0000 7fa9dbf1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:53.112 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:53.110+0000 7fa9dbf1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:53.112 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:53.111+0000 7fa9dbf1e440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:43:53.112 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:53.111+0000 7fa9dbf1e440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:43:53.112 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:53.111+0000 7fa9dbf1e440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:43:53.189 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:53.187+0000 7fa9dbf1e440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-18T11:43:59.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-18T11:43:59.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:43:59.108 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T11:43:59.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:43:59.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:43:59.593 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T11:43:59.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:43:59.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:43:59.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:43:59.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:43:59.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:43:59.616 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:59.614+0000 7f0292f7d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:59.617 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:59.616+0000 7f0292f7d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:59.618 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:43:59.617+0000 7f0292f7d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:59.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:43:59.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:43:59.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:43:59.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:43:59.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:43:59.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:59.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:43:59.947 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:43:59.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:59.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:00.135 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:44:00.134+0000 7f0292f7d440 -1 Falling back to public interface 2024-09-18T11:44:00.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:01.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:01.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:01.297 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:44:01.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:44:01.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:01.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:01.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:02.633 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:44:02.631+0000 7f0292f7d440 -1 osd.1 0 log_to_monitors true 2024-09-18T11:44:02.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:02.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:02.658 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:44:02.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:44:02.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:02.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:03.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:04.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:04.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:04.093 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:44:04.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:44:04.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:04.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:04.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:05.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:05.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:05.561 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:44:05.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:44:05.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:05.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:06.035 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:07.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:07.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:07.037 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:44:07.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:44:07.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:07.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:07.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:08.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:08.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:08.495 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:44:08.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:44:08.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:08.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:08.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:09.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:09.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:09.893 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:44:09.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:44:09.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:09.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:10.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:11.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:11.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:11.426 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:44:11.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:44:11.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:11.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:11.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:12.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:12.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:12.908 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:44:12.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:44:12.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:12.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:13.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:14.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:14.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:14.449 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:44:14.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:44:14.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:14.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:14.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:15.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:15.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:15.853 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:44:15.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:44:15.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:15.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:17.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:18.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:18.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:18.009 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:44:18.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:44:18.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:18.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:18.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:19.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:19.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:19.720 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:44:19.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:44:19.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:19.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:20.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:21.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:21.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:21.397 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:44:21.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:44:21.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:21.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:22.268 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:23.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:23.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:23.270 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:44:23.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:44:23.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:23.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:24.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:25.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:25.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:25.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:44:25.066 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:44:25.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:25.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:25.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:26.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:26.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:26.573 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:44:26.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:44:26.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:26.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:26.932 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/113688,v1:127.0.0.1:6811/113688] [v2:127.0.0.1:6812/113688,v1:127.0.0.1:6813/113688] exists,up cf4898a4-6ad1-4687-891b-571e82dd3d2f 2024-09-18T11:44:26.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:44:26.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:44:26.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:44:26.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: test_get_osds: create_rbd_pool 2024-09-18T11:44:26.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:44:27.297 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:44:27.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:44:27.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:44:27.913 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:44:27.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:44:28.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:44:30.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1039: test_get_osds: wait_for_clean 2024-09-18T11:44:30.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:44:30.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:44:30.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:44:30.711 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:44:30.711 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:44:30.711 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:44:30.711 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:44:30.712 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:44:30.712 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:44:30.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:44:30.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:44:30.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:44:30.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:44:30.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:44:30.808 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:44:31.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:44:31.159 INFO:tasks.workunit.client.0.smithi157.stderr:1' 2024-09-18T11:44:31.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:44:31.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:44:31.159 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:44:31.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836494 2024-09-18T11:44:31.267 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836494 2024-09-18T11:44:31.267 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836494' 2024-09-18T11:44:31.267 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:44:31.267 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:44:31.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672966 2024-09-18T11:44:31.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672966 2024-09-18T11:44:31.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836494 1-42949672966' 2024-09-18T11:44:31.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:44:31.385 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836494 2024-09-18T11:44:31.385 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:44:31.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:44:31.386 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836494 2024-09-18T11:44:31.386 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:44:31.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836494 2024-09-18T11:44:31.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-09-18T11:44:31.387 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836494 2024-09-18T11:44:31.387 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:44:31.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836492 -lt 21474836494 2024-09-18T11:44:31.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:44:32.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:44:32.736 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:44:33.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836492 -lt 21474836494 2024-09-18T11:44:33.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:44:34.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T11:44:34.103 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:44:34.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836495 -lt 21474836494 2024-09-18T11:44:34.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:44:34.493 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672966 2024-09-18T11:44:34.493 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:44:34.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:44:34.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672966 2024-09-18T11:44:34.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:44:34.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672966 2024-09-18T11:44:34.495 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672966 2024-09-18T11:44:34.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-09-18T11:44:34.496 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:44:34.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672966 -lt 42949672966 2024-09-18T11:44:34.903 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:44:34.903 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:44:34.903 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:44:35.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:44:35.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:44:35.373 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:44:35.373 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:44:35.374 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:44:35.374 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:44:35.374 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:44:35.374 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:44:35.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:44:35.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:44:35.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:44:35.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:44:36.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T11:44:36.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:44:36.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:44:36.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1040: test_get_osds: create_rbd_pool 2024-09-18T11:44:36.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:44:36.717 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:44:36.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:44:36.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:44:37.412 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:44:37.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:44:38.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:44:40.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: get_osds rbd GROUP 2024-09-18T11:44:40.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=rbd 2024-09-18T11:44:40.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2024-09-18T11:44:40.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=GROUP 2024-09-18T11:44:40.118 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map rbd GROUP 2024-09-18T11:44:40.118 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:44:40.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-18T11:44:40.484 INFO:tasks.workunit.client.0.smithi157.stderr:0' 2024-09-18T11:44:40.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-09-18T11:44:40.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1042: test_get_osds: teardown td/ceph-helpers 2024-09-18T11:44:40.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:44:40.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:44:40.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:44:40.485 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:44:40.485 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:44:40.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:44:40.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:44:40.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:44:40.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:44:40.591 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:44:40.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:44:40.591 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:44:40.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:44:40.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:44:40.593 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:44:40.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:44:40.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:44:40.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:44:40.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:44:40.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:44:40.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:44:40.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:44:40.606 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:44:40.606 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:44:40.606 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:44:40.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:44:40.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:44:40.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:44:40.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:44:40.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_pg td/ceph-helpers 2024-09-18T11:44:40.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: local dir=td/ceph-helpers 2024-09-18T11:44:40.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: test_get_pg: setup td/ceph-helpers 2024-09-18T11:44:40.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:44:40.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:44:40.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:44:40.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:44:40.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:44:40.609 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:44:40.609 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:44:40.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:44:40.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:44:40.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:44:40.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:44:40.610 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:44:40.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:44:40.610 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:44:40.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:44:40.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:44:40.611 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:44:40.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:44:40.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:44:40.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:44:40.613 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:44:40.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:44:40.614 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:44:40.614 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:44:40.615 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:44:40.615 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:44:40.615 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:44:40.615 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:44:40.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:44:40.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:44:40.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:44:40.617 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:44:40.617 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:44:40.617 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:44:40.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:44:40.619 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:44:40.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:44:40.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:44:40.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:44:40.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:44:40.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:44:40.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:44:40.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:44:40.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:44:40.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:44:40.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:44:41.004 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:44:41.005 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:44:41.005 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:44:41.005 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:44:41.005 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:44:41.005 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:44:41.005 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:44:41.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:44:41.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:44:41.241 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:44:41.241 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:44:41.242 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:44:41.242 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:44:41.242 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:44:41.242 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:44:41.242 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:44:41.242 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:44:41.242 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:44:41.242 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:44:41.242 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:44:41.242 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:44:41.242 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:44:41.242 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:44:41.317 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:44:41.318 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:44:41.318 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:44:41.318 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:44:41.318 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:44:41.318 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:44:41.318 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:44:41.318 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:44:41.318 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:44:41.318 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:44:41.318 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:44:41.319 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:44:41.319 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:44:41.319 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:44:41.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: test_get_pg: run_mgr td/ceph-helpers x 2024-09-18T11:44:41.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:44:41.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:44:41.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:44:41.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:44:41.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:44:41.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:44:42.226 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:44:42.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:44:42.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:44:42.227 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:44:42.227 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:44:42.227 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:44:42.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:44:42.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:44:42.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:44:42.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: test_get_pg: run_osd td/ceph-helpers 0 2024-09-18T11:44:42.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:44:42.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:44:42.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:44:42.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:44:42.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:44:42.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:44:42.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:44:42.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:44:42.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:44:42.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:44:42.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:44:42.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:44:42.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:44:42.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:44:42.255 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:44:42.256 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:44:42.256 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:44:42.256 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:44:42.256 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:44:42.256 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:44:42.256 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:44:42.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:44:42.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:44:42.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:44:42.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:44:42.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:44:42.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:44:42.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:44:42.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:44:42.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:44:42.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:44:42.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:44:42.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:44:42.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:44:42.257 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:44:42.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=69c77ccc-26ec-4b4b-9cf4-ca38cf9c94be 2024-09-18T11:44:42.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 69c77ccc-26ec-4b4b-9cf4-ca38cf9c94be' 2024-09-18T11:44:42.258 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 69c77ccc-26ec-4b4b-9cf4-ca38cf9c94be 2024-09-18T11:44:42.258 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:44:42.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAqvepmEnAEEBAAly7Zy7thraklMPgUqLgGUA== 2024-09-18T11:44:42.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAqvepmEnAEEBAAly7Zy7thraklMPgUqLgGUA=="}' 2024-09-18T11:44:42.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 69c77ccc-26ec-4b4b-9cf4-ca38cf9c94be -i td/ceph-helpers/0/new.json 2024-09-18T11:44:42.579 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:44:42.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:44:42.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAqvepmEnAEEBAAly7Zy7thraklMPgUqLgGUA== --osd-uuid 69c77ccc-26ec-4b4b-9cf4-ca38cf9c94be 2024-09-18T11:44:42.614 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:44:42.612+0000 7f483e41f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:44:42.625 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:44:42.623+0000 7f483e41f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:44:42.626 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:44:42.624+0000 7f483e41f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:44:42.626 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:44:42.625+0000 7f483e41f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:44:42.626 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:44:42.625+0000 7f483e41f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:44:42.626 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:44:42.625+0000 7f483e41f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:44:42.664 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:44:42.663+0000 7f483e41f440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:44:48.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:44:48.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:44:48.071 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:44:48.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:44:48.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:44:48.559 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:44:48.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:44:48.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:44:48.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:44:48.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:44:48.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:44:48.582 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:44:48.580+0000 7fa36fc4c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:44:48.583 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:44:48.582+0000 7fa36fc4c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:44:48.585 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:44:48.583+0000 7fa36fc4c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:44:48.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:44:48.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:44:48.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:44:48.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:44:48.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:44:48.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:48.911 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:44:48.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:44:48.912 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:48.912 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:44:49.116 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:44:49.115+0000 7fa36fc4c440 -1 Falling back to public interface 2024-09-18T11:44:49.267 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:50.268 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:50.268 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:50.269 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:44:50.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:44:50.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:50.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:44:50.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:51.590 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:44:51.589+0000 7fa36fc4c440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:44:51.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:51.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:51.678 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:44:51.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:44:51.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:51.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:44:52.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:53.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:53.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:53.143 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:44:53.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:44:53.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:53.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:44:53.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:54.562 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:54.562 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:54.563 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:44:54.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:44:54.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:54.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:44:54.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:55.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:55.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:55.926 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:44:55.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:44:55.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:55.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:44:56.371 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:57.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:57.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:57.373 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:44:57.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:44:57.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:57.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:44:57.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:58.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:58.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:58.754 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:44:58.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:44:58.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:58.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:44:59.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:00.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:00.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:00.122 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:45:00.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:45:00.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:00.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:00.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:01.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:01.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:01.573 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:45:01.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:45:01.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:01.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:01.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:02.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:02.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:02.964 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:45:02.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:45:02.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:02.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:03.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:04.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:04.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:04.639 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:45:04.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:45:04.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:04.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:05.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:06.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:06.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:06.320 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:45:06.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:45:06.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:06.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:06.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:07.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:07.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:07.931 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:45:07.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:45:07.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:07.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:08.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:09.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:09.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:09.508 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:45:09.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:45:09.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:09.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:10.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:11.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:11.168 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:45:11.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:11.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:45:11.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:11.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:11.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:12.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:12.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:12.938 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:45:12.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:45:12.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:12.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:13.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:14.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:14.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:14.609 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:45:14.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:45:14.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:14.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:14.980 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/116174,v1:127.0.0.1:6803/116174] [v2:127.0.0.1:6804/116174,v1:127.0.0.1:6805/116174] exists,up 69c77ccc-26ec-4b4b-9cf4-ca38cf9c94be 2024-09-18T11:45:14.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:45:14.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:45:14.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:45:14.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: test_get_pg: create_rbd_pool 2024-09-18T11:45:14.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:45:15.361 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:45:15.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:45:15.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:45:15.899 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:45:15.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:45:16.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:45:17.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: test_get_pg: wait_for_clean 2024-09-18T11:45:17.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:45:17.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:45:17.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:45:17.891 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:45:17.891 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:45:17.892 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:45:17.892 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:45:17.892 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:45:17.892 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:45:17.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:45:17.988 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:45:17.988 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:45:17.988 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:45:17.988 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:45:17.988 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:45:18.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T11:45:18.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:45:18.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:45:18.333 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:45:18.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836484 2024-09-18T11:45:18.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836484 2024-09-18T11:45:18.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836484' 2024-09-18T11:45:18.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:45:18.442 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836484 2024-09-18T11:45:18.442 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:45:18.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:45:18.443 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836484 2024-09-18T11:45:18.443 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:45:18.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836484 2024-09-18T11:45:18.444 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836484 2024-09-18T11:45:18.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-09-18T11:45:18.444 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:45:18.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836482 -lt 21474836484 2024-09-18T11:45:18.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:45:19.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:45:19.798 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:45:20.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836484 2024-09-18T11:45:20.153 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:45:20.154 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:45:20.154 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:45:20.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:45:20.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:45:20.662 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:45:20.662 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:45:20.662 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:45:20.663 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:45:20.663 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:45:20.663 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:45:21.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:45:21.003 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:45:21.003 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:45:21.003 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:45:21.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T11:45:21.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:45:21.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:45:21.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: get_pg rbd GROUP 2024-09-18T11:45:21.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-09-18T11:45:21.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2024-09-18T11:45:21.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=GROUP 2024-09-18T11:45:21.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd GROUP 2024-09-18T11:45:21.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T11:45:21.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1109: test_get_pg: teardown td/ceph-helpers 2024-09-18T11:45:21.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:45:21.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:45:21.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:45:21.894 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:45:21.894 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:45:21.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:45:21.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:45:21.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:45:21.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:45:21.999 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:45:22.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:45:22.000 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:45:22.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:45:22.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:45:22.002 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:45:22.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:45:22.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:45:22.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:45:22.003 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:45:22.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:45:22.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:45:22.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:45:22.012 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:45:22.012 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:45:22.012 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:45:22.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:45:22.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:45:22.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:45:22.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:45:22.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_primary td/ceph-helpers 2024-09-18T11:45:22.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: local dir=td/ceph-helpers 2024-09-18T11:45:22.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: test_get_primary: setup td/ceph-helpers 2024-09-18T11:45:22.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:45:22.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:45:22.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:45:22.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:45:22.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:45:22.014 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:45:22.014 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:45:22.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:45:22.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:45:22.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:45:22.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:45:22.015 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:45:22.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:45:22.015 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:45:22.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:45:22.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:45:22.016 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:45:22.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:45:22.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:45:22.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:45:22.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:45:22.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:45:22.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:45:22.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:45:22.020 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:45:22.020 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:45:22.020 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:45:22.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:45:22.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:45:22.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:45:22.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:45:22.022 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:45:22.022 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:45:22.022 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:45:22.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:45:22.024 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:45:22.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:45:22.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:45:22.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:45:22.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:45:22.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:45:22.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:45:22.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:45:22.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:45:22.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:45:22.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:45:22.359 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:45:22.359 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:45:22.360 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:45:22.360 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:45:22.360 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:45:22.360 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:45:22.360 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:45:22.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:45:22.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:45:22.612 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:45:22.612 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:45:22.612 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:45:22.612 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:45:22.612 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:45:22.612 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:45:22.612 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:45:22.612 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:45:22.612 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:45:22.612 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:45:22.612 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:45:22.612 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:45:22.613 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:45:22.613 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:45:22.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:45:22.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:45:22.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:45:22.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:45:22.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:45:22.688 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:45:22.688 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:45:22.688 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:45:22.688 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:45:22.688 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:45:22.689 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:45:22.689 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:45:22.689 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:45:22.689 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:45:22.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: test_get_primary: local osd=0 2024-09-18T11:45:22.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: test_get_primary: run_mgr td/ceph-helpers x 2024-09-18T11:45:22.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:45:22.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:45:22.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:45:22.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:45:22.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:45:22.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:45:23.595 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:45:23.595 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:45:23.595 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:45:23.596 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:45:23.596 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:45:23.596 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:45:23.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:45:23.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:45:23.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:45:23.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: test_get_primary: run_osd td/ceph-helpers 0 2024-09-18T11:45:23.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:45:23.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:45:23.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:45:23.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:45:23.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:45:23.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:45:23.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:45:23.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:45:23.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:45:23.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:45:23.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:45:23.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:45:23.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:45:23.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:45:23.625 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:45:23.625 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:45:23.625 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:45:23.625 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:45:23.625 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:45:23.625 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:45:23.625 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:45:23.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:45:23.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:45:23.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:45:23.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:45:23.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:45:23.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:45:23.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:45:23.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:45:23.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:45:23.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:45:23.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:45:23.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:45:23.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:45:23.626 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:45:23.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c3e8c4d6-bd55-4d29-83e2-d53b75cacff1 2024-09-18T11:45:23.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c3e8c4d6-bd55-4d29-83e2-d53b75cacff1' 2024-09-18T11:45:23.627 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 c3e8c4d6-bd55-4d29-83e2-d53b75cacff1 2024-09-18T11:45:23.628 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:45:23.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTvepmi6//JRAArf8ei8xtGeLFWGanu6Skow== 2024-09-18T11:45:23.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTvepmi6//JRAArf8ei8xtGeLFWGanu6Skow=="}' 2024-09-18T11:45:23.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c3e8c4d6-bd55-4d29-83e2-d53b75cacff1 -i td/ceph-helpers/0/new.json 2024-09-18T11:45:23.965 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:45:23.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:45:23.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTvepmi6//JRAArf8ei8xtGeLFWGanu6Skow== --osd-uuid c3e8c4d6-bd55-4d29-83e2-d53b75cacff1 2024-09-18T11:45:24.000 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:45:23.999+0000 7f394640b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:45:24.012 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:45:24.010+0000 7f394640b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:45:24.013 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:45:24.012+0000 7f394640b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:45:24.013 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:45:24.012+0000 7f394640b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:45:24.014 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:45:24.012+0000 7f394640b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:45:24.014 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:45:24.012+0000 7f394640b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:45:24.052 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:45:24.050+0000 7f394640b440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:45:29.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:45:29.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:45:29.659 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:45:29.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:45:29.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:45:30.139 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:45:30.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:45:30.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:45:30.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:45:30.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:45:30.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:45:30.161 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:45:30.160+0000 7fe678d2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:45:30.163 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:45:30.162+0000 7fe678d2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:45:30.164 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:45:30.163+0000 7fe678d2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:45:30.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:45:30.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:45:30.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:45:30.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:45:30.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:45:30.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:30.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:45:30.488 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:45:30.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:30.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:30.694 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:45:30.693+0000 7fe678d2d440 -1 Falling back to public interface 2024-09-18T11:45:30.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:31.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:31.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:31.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:45:31.848 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:45:31.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:31.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:32.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:33.201 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:45:33.200+0000 7fe678d2d440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:45:33.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:33.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:33.208 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:45:33.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:45:33.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:33.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:33.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:34.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:34.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:34.595 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:45:34.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:45:34.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:34.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:35.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:36.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:36.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:36.040 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:45:36.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:45:36.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:36.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:36.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:37.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:37.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:37.420 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:45:37.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:45:37.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:37.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:37.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:38.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:38.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:38.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:45:38.924 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:45:38.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:38.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:39.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:40.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:40.289 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:40.289 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:45:40.289 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:45:40.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:40.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:40.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:41.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:41.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:41.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:45:41.685 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:45:41.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:41.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:42.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:43.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:43.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:43.066 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:45:43.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:45:43.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:43.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:43.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:44.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:44.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:44.472 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:45:44.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:45:44.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:44.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:44.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:45.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:45.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:45.842 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:45:45.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:45:45.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:45.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:46.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:47.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:47.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:47.576 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:45:47.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:45:47.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:47.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:48.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:49.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:49.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:49.451 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:45:49.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:45:49.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:49.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:50.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:51.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:51.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:51.454 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:45:51.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:45:51.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:51.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:52.500 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:53.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:53.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:53.501 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:45:53.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:45:53.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:53.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:54.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:55.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:55.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:55.660 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:45:55.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:45:55.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:55.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:45:56.099 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/118513,v1:127.0.0.1:6803/118513] [v2:127.0.0.1:6804/118513,v1:127.0.0.1:6805/118513] exists,up c3e8c4d6-bd55-4d29-83e2-d53b75cacff1 2024-09-18T11:45:56.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:45:56.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:45:56.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:45:56.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: test_get_primary: create_rbd_pool 2024-09-18T11:45:56.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:45:56.507 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:45:56.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:45:56.518 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:45:56.997 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:45:57.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:45:58.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:45:59.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: test_get_primary: wait_for_clean 2024-09-18T11:45:59.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:45:59.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:45:59.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:45:59.186 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:45:59.187 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:45:59.187 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:45:59.187 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:45:59.187 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:45:59.187 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:45:59.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:45:59.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:45:59.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:45:59.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:45:59.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:45:59.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:45:59.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T11:45:59.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:45:59.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:45:59.632 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:45:59.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836485 2024-09-18T11:45:59.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836485 2024-09-18T11:45:59.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485' 2024-09-18T11:45:59.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:45:59.740 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836485 2024-09-18T11:45:59.740 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:45:59.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:45:59.741 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836485 2024-09-18T11:45:59.741 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:45:59.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836485 2024-09-18T11:45:59.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-18T11:45:59.742 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836485 2024-09-18T11:45:59.743 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:46:00.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836483 -lt 21474836485 2024-09-18T11:46:00.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:46:01.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:46:01.102 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:46:01.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836485 2024-09-18T11:46:01.473 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:46:01.474 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:46:01.474 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:46:01.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:46:01.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:46:01.959 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:46:01.959 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:46:01.959 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:46:01.959 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:46:01.959 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:46:01.960 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:46:02.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:46:02.301 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:46:02.302 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:46:02.302 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:46:02.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T11:46:02.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:46:02.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:46:02.770 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: get_primary rbd GROUP 2024-09-18T11:46:02.771 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-09-18T11:46:02.771 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-09-18T11:46:02.771 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-09-18T11:46:02.771 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T11:46:03.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: test 0 = 0 2024-09-18T11:46:03.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: test_get_primary: teardown td/ceph-helpers 2024-09-18T11:46:03.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:46:03.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:46:03.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:46:03.202 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:46:03.203 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:46:03.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:46:03.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:46:03.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:46:03.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:46:03.308 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:46:03.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:46:03.309 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:46:03.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:46:03.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:46:03.310 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:46:03.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:46:03.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:46:03.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:46:03.311 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:46:03.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:46:03.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:46:03.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:46:03.319 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:46:03.319 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:03.319 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:46:03.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:46:03.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:46:03.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:46:03.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:46:03.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_get_timeout_delays td/ceph-helpers 2024-09-18T11:46:03.320 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: get_timeout_delays 1 2024-09-18T11:46:03.321 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:46:03.321 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:46:03.321 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:46:03.321 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:46:03.321 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:46:03.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: test '1 ' = '1 ' 2024-09-18T11:46:03.331 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: get_timeout_delays 5 2024-09-18T11:46:03.332 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:46:03.332 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:46:03.332 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:46:03.332 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:46:03.332 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:46:03.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2024-09-18T11:46:03.350 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: get_timeout_delays 6 2024-09-18T11:46:03.351 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:46:03.351 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:46:03.351 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:46:03.351 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:46:03.351 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:46:03.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2024-09-18T11:46:03.369 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: get_timeout_delays 7 2024-09-18T11:46:03.370 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:46:03.370 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:46:03.370 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:46:03.370 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:46:03.370 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:46:03.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2024-09-18T11:46:03.394 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: get_timeout_delays 8 2024-09-18T11:46:03.394 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:46:03.394 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:46:03.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:46:03.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:46:03.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:46:03.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2024-09-18T11:46:03.420 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: get_timeout_delays 1 .1 2024-09-18T11:46:03.420 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:46:03.420 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:46:03.420 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:46:03.420 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:46:03.421 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:46:03.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2024-09-18T11:46:03.446 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: get_timeout_delays 1.5 .1 2024-09-18T11:46:03.446 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:46:03.446 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:46:03.446 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:46:03.446 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:46:03.446 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:46:03.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2024-09-18T11:46:03.476 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: get_timeout_delays 5 .1 2024-09-18T11:46:03.477 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:46:03.477 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:46:03.478 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:46:03.478 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:46:03.478 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:46:03.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2024-09-18T11:46:03.516 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: get_timeout_delays 6 .1 2024-09-18T11:46:03.517 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:46:03.517 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:46:03.517 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:46:03.517 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:46:03.517 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:46:03.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2024-09-18T11:46:03.555 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: get_timeout_delays 6.3 .1 2024-09-18T11:46:03.556 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:46:03.556 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:46:03.556 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:46:03.556 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:46:03.556 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:46:03.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2024-09-18T11:46:03.595 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: get_timeout_delays 20 .1 2024-09-18T11:46:03.595 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:46:03.595 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:46:03.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:46:03.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:46:03.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:46:03.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2024-09-18T11:46:03.647 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: get_timeout_delays 300 .1 0 2024-09-18T11:46:03.648 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:46:03.648 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:46:03.648 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:46:03.648 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:46:03.648 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:46:03.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2024-09-18T11:46:03.726 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: get_timeout_delays 300 .1 10 2024-09-18T11:46:03.727 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:46:03.727 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:46:03.727 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:46:03.727 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:46:03.727 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:46:03.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2024-09-18T11:46:03.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:46:03.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_is_clean td/ceph-helpers 2024-09-18T11:46:03.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: local dir=td/ceph-helpers 2024-09-18T11:46:03.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: test_is_clean: setup td/ceph-helpers 2024-09-18T11:46:03.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:46:03.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:46:03.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:46:03.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:46:03.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:46:03.966 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:46:03.966 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:46:03.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:46:03.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:46:03.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:46:03.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:46:03.967 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:46:03.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:46:03.967 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:46:03.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:46:03.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:46:03.968 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:46:03.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:46:03.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:46:03.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:46:03.970 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:46:03.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:46:03.971 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:46:03.971 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:46:03.972 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:46:03.972 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:03.972 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:46:03.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:46:03.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:46:03.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:46:03.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:46:03.974 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:46:03.974 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:03.974 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:46:03.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:46:03.976 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:46:03.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:46:03.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:46:03.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:46:03.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:46:03.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:46:03.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:46:03.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:46:03.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:46:03.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:46:03.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:46:04.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:46:04.347 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:46:04.347 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:46:04.347 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:46:04.347 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:04.347 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:46:04.347 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:46:04.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:46:04.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:46:04.608 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:46:04.608 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:46:04.608 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:46:04.608 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:46:04.608 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:46:04.608 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:46:04.608 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:46:04.609 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:46:04.609 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:46:04.609 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:04.609 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:46:04.609 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:46:04.609 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:46:04.609 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:46:04.683 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:46:04.683 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:46:04.683 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:46:04.683 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:46:04.684 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:46:04.684 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:46:04.684 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:46:04.684 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:46:04.684 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:46:04.684 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:04.684 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:46:04.684 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:46:04.684 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:46:04.684 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:46:04.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: test_is_clean: run_mgr td/ceph-helpers x 2024-09-18T11:46:04.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:46:04.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:46:04.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:46:04.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:46:04.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:46:04.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:46:05.593 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:46:05.593 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:46:05.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:46:05.594 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:46:05.594 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:05.594 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:46:05.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:46:05.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:46:05.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:46:05.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: test_is_clean: run_osd td/ceph-helpers 0 2024-09-18T11:46:05.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:46:05.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:46:05.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:46:05.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:46:05.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:46:05.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:46:05.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:46:05.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:46:05.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:46:05.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:46:05.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:46:05.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:46:05.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:46:05.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:46:05.623 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:46:05.623 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:46:05.623 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:46:05.623 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:46:05.623 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:05.623 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:46:05.623 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:46:05.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:46:05.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:46:05.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:46:05.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:46:05.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:46:05.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:46:05.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:46:05.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:46:05.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:46:05.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:46:05.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:46:05.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:46:05.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:46:05.625 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:46:05.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=58773740-c1b5-49ec-912c-215b39600b5e 2024-09-18T11:46:05.625 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 58773740-c1b5-49ec-912c-215b39600b5e 2024-09-18T11:46:05.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 58773740-c1b5-49ec-912c-215b39600b5e' 2024-09-18T11:46:05.626 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:46:05.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9vepmK2DgJRAASVPUy0OIVMWrfHn+UecU1w== 2024-09-18T11:46:05.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9vepmK2DgJRAASVPUy0OIVMWrfHn+UecU1w=="}' 2024-09-18T11:46:05.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 58773740-c1b5-49ec-912c-215b39600b5e -i td/ceph-helpers/0/new.json 2024-09-18T11:46:05.937 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:46:05.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:46:05.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9vepmK2DgJRAASVPUy0OIVMWrfHn+UecU1w== --osd-uuid 58773740-c1b5-49ec-912c-215b39600b5e 2024-09-18T11:46:05.971 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:05.970+0000 7f4c67524440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:46:05.983 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:05.981+0000 7f4c67524440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:46:05.984 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:05.983+0000 7f4c67524440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:46:05.984 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:05.983+0000 7f4c67524440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:46:05.984 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:05.983+0000 7f4c67524440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:46:05.984 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:05.983+0000 7f4c67524440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:46:06.023 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:06.021+0000 7f4c67524440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:46:11.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:46:11.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:46:11.388 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:46:11.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:46:11.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:46:11.878 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:46:11.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:46:11.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:46:11.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:46:11.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:46:11.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:46:11.900 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:11.899+0000 7faf42018440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:46:11.902 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:11.901+0000 7faf42018440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:46:11.903 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:11.902+0000 7faf42018440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:46:12.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:46:12.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:46:12.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:46:12.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:46:12.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:46:12.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:12.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:46:12.274 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:46:12.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:12.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:12.420 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:12.419+0000 7faf42018440 -1 Falling back to public interface 2024-09-18T11:46:12.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:13.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:13.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:13.626 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:46:13.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:46:13.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:13.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:13.985 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:14.946 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:14.945+0000 7faf42018440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:46:14.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:14.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:14.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:46:14.987 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:46:14.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:14.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:15.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:16.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:16.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:16.566 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:46:16.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:46:16.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:16.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:16.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:17.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:17.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:17.969 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:46:17.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:46:17.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:17.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:18.375 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:19.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:19.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:19.377 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:46:19.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:46:19.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:19.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:19.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:20.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:20.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:20.802 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:46:20.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:46:20.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:20.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:21.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:22.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:22.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:22.169 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:46:22.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:46:22.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:22.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:22.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:23.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:23.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:23.725 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:46:23.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:46:23.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:23.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:24.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:25.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:25.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:25.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:46:25.104 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:46:25.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:25.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:25.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:26.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:26.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:26.467 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:46:26.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:46:26.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:26.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:27.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:28.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:28.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:28.132 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:46:28.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:46:28.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:28.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:28.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:29.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:29.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:29.925 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:46:29.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:46:29.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:29.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:30.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:31.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:31.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:31.675 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:46:31.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:46:31.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:31.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:32.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:33.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:33.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:33.516 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:46:33.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:46:33.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:33.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:34.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:35.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:35.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:35.240 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:46:35.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:46:35.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:35.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:35.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:36.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:36.720 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:46:36.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:36.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:46:36.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:36.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:37.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:38.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:38.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:38.266 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:46:38.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:46:38.267 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:38.267 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:38.622 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/121233,v1:127.0.0.1:6803/121233] [v2:127.0.0.1:6804/121233,v1:127.0.0.1:6805/121233] exists,up 58773740-c1b5-49ec-912c-215b39600b5e 2024-09-18T11:46:38.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:46:38.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:46:38.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:46:38.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: test_is_clean: create_rbd_pool 2024-09-18T11:46:38.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:46:38.979 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:46:38.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:46:38.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:46:39.468 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:46:39.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:46:40.480 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:46:41.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: test_is_clean: wait_for_clean 2024-09-18T11:46:41.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:46:41.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:46:41.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:46:41.621 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:46:41.621 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:46:41.622 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:46:41.622 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:46:41.622 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:46:41.622 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:46:41.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:46:41.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:46:41.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:46:41.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:46:41.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:46:41.717 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:46:42.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T11:46:42.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:46:42.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:46:42.068 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:46:42.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836489 2024-09-18T11:46:42.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836489 2024-09-18T11:46:42.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836489' 2024-09-18T11:46:42.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:46:42.177 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836489 2024-09-18T11:46:42.177 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:46:42.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:46:42.178 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836489 2024-09-18T11:46:42.178 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:46:42.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836489 2024-09-18T11:46:42.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-09-18T11:46:42.179 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836489 2024-09-18T11:46:42.179 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:46:42.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836487 -lt 21474836489 2024-09-18T11:46:42.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:46:43.534 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:46:43.534 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:46:43.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836489 -lt 21474836489 2024-09-18T11:46:43.932 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:46:43.932 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:46:43.932 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:46:44.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:46:44.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:46:44.415 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:46:44.415 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:46:44.415 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:46:44.415 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:46:44.415 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:46:44.415 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:46:44.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:46:44.759 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:46:44.759 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:46:44.759 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:46:45.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T11:46:45.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:46:45.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:46:45.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1569: test_is_clean: is_clean 2024-09-18T11:46:45.233 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: get_num_pgs 2024-09-18T11:46:45.233 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:46:45.233 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:46:45.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: num_pgs=4 2024-09-18T11:46:45.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: is_clean: test 4 '!=' 0 2024-09-18T11:46:45.732 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: get_num_active_clean 2024-09-18T11:46:45.732 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:46:45.732 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:46:45.732 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:46:45.732 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:46:45.733 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:46:46.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: test 4 = 4 2024-09-18T11:46:46.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: test_is_clean: teardown td/ceph-helpers 2024-09-18T11:46:46.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:46:46.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:46:46.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:46:46.074 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:46:46.074 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:46:46.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:46:46.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:46:46.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:46:46.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:46:46.180 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:46:46.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:46:46.181 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:46:46.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:46:46.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:46:46.182 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:46:46.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:46:46.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:46:46.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:46:46.183 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:46:46.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:46:46.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:46:46.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:46:46.191 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:46:46.191 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:46.191 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:46:46.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:46:46.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:46:46.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:46:46.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:46:46.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_kill_daemon td/ceph-helpers 2024-09-18T11:46:46.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:275: test_kill_daemon: local dir=td/ceph-helpers 2024-09-18T11:46:46.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:276: test_kill_daemon: setup td/ceph-helpers 2024-09-18T11:46:46.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:46:46.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:46:46.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:46:46.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:46:46.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:46:46.194 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:46:46.194 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:46:46.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:46:46.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:46:46.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:46:46.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:46:46.194 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:46:46.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:46:46.195 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:46:46.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:46:46.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:46:46.196 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:46:46.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:46:46.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:46:46.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:46:46.198 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:46:46.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:46:46.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:46:46.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:46:46.201 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:46:46.201 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:46.201 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:46:46.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:46:46.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:46:46.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:46:46.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:46:46.203 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:46:46.203 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:46.203 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:46:46.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:46:46.205 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:46:46.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:46:46.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:46:46.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:46:46.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:277: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:46:46.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:46:46.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:46:46.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:46:46.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:46:46.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:46:46.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:46:46.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:46:46.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:46:46.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:46:46.565 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:46:46.565 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:46.565 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:46:46.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:46:46.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:46:46.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:46:46.834 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:46:46.834 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:46:46.834 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:46:46.834 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:46:46.834 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:46:46.834 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:46:46.834 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:46:46.834 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:46:46.835 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:46:46.835 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:46.835 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:46:46.835 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:46:46.835 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:46:46.835 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:46:46.912 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:46:46.912 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:46:46.912 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:46:46.913 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:46:46.913 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:46:46.913 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:46:46.913 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:46:46.913 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:46:46.913 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:46:46.913 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:46.914 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:46:46.914 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:46:46.914 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:46:46.914 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:46:46.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:278: test_kill_daemon: run_mgr td/ceph-helpers x 2024-09-18T11:46:46.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:46:46.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:46:46.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:46:46.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:46:46.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:46:46.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:46:47.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:46:47.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:46:47.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:46:47.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:46:47.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:47.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:46:47.821 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:46:47.821 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:46:47.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:46:47.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:279: test_kill_daemon: run_osd td/ceph-helpers 0 2024-09-18T11:46:47.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:46:47.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:46:47.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:46:47.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:46:47.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:46:47.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:46:47.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:46:47.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:46:47.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:46:47.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:46:47.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:46:47.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:46:47.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:46:47.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:46:47.848 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:46:47.848 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:46:47.848 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:46:47.848 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:46:47.848 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:47.848 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:46:47.848 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:46:47.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:46:47.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:46:47.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:46:47.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:46:47.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:46:47.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:46:47.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:46:47.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:46:47.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:46:47.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:46:47.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:46:47.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:46:47.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:46:47.849 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:46:47.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=755ada38-a6c4-40af-b07c-002797db9a83 2024-09-18T11:46:47.850 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 755ada38-a6c4-40af-b07c-002797db9a83 2024-09-18T11:46:47.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 755ada38-a6c4-40af-b07c-002797db9a83' 2024-09-18T11:46:47.851 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:46:47.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnvepmuhRXMxAAX9rctOg4hX2juDl3/JOjPw== 2024-09-18T11:46:47.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnvepmuhRXMxAAX9rctOg4hX2juDl3/JOjPw=="}' 2024-09-18T11:46:47.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 755ada38-a6c4-40af-b07c-002797db9a83 -i td/ceph-helpers/0/new.json 2024-09-18T11:46:48.171 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:46:48.181 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:46:48.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCnvepmuhRXMxAAX9rctOg4hX2juDl3/JOjPw== --osd-uuid 755ada38-a6c4-40af-b07c-002797db9a83 2024-09-18T11:46:48.206 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:48.205+0000 7fe782009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:46:48.226 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:48.224+0000 7fe782009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:46:48.227 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:48.226+0000 7fe782009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:46:48.227 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:48.226+0000 7fe782009440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:46:48.227 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:48.226+0000 7fe782009440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:46:48.227 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:48.226+0000 7fe782009440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:46:48.266 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:48.265+0000 7fe782009440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:46:53.583 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:46:53.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:46:53.584 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:46:53.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:46:53.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:46:54.076 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:46:54.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:46:54.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:46:54.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:46:54.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:46:54.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:46:54.097 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:54.096+0000 7efe9bd1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:46:54.100 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:54.098+0000 7efe9bd1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:46:54.100 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:54.099+0000 7efe9bd1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:46:54.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:46:54.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:46:54.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:46:54.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:46:54.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:46:54.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:54.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:46:54.429 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:46:54.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:54.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:54.625 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:54.624+0000 7efe9bd1f440 -1 Falling back to public interface 2024-09-18T11:46:54.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:55.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:55.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:55.812 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:46:55.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:46:55.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:55.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:56.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:57.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:57.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:57.175 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:46:57.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:46:57.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:57.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:57.176 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:46:57.174+0000 7efe9bd1f440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:46:57.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:58.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:58.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:58.543 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:46:58.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:46:58.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:58.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:46:58.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:59.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:59.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:59.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:46:59.947 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:46:59.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:59.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:00.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:01.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:01.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:01.352 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:47:01.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:47:01.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:01.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:01.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:02.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:02.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:02.860 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:47:02.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:47:02.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:02.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:03.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:04.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:04.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:04.223 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:47:04.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:47:04.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:04.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:04.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:05.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:05.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:05.601 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:47:05.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:47:05.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:05.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:06.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:07.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:07.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:07.015 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:47:07.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:47:07.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:07.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:07.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:08.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:08.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:08.416 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:47:08.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:47:08.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:08.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:08.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:09.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:09.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:09.774 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:47:09.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:47:09.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:09.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:10.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:11.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:11.214 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:47:11.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:11.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:47:11.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:11.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:11.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:12.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:12.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:12.699 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:47:12.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:47:12.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:12.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:13.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:14.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:14.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:14.430 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:47:14.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:47:14.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:14.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:15.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:16.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:16.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:16.199 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:47:16.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:47:16.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:16.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:17.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:18.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:18.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:18.046 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:47:18.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:47:18.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:18.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:18.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:19.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:19.828 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:19.828 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:47:19.828 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:47:19.828 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:19.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:20.196 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/123597,v1:127.0.0.1:6803/123597] [v2:127.0.0.1:6804/123597,v1:127.0.0.1:6805/123597] exists,up 755ada38-a6c4-40af-b07c-002797db9a83 2024-09-18T11:47:20.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:47:20.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:47:20.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:47:20.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:281: test_kill_daemon: name_prefix=osd 2024-09-18T11:47:20.198 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: find td/ceph-helpers 2024-09-18T11:47:20.199 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: grep 'osd[^/]*\.pid' 2024-09-18T11:47:20.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-18T11:47:20.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:289: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2024-09-18T11:47:20.202 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-09-18T11:47:20.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=123597 2024-09-18T11:47:20.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=0 2024-09-18T11:47:20.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local delays=1 2024-09-18T11:47:20.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-09-18T11:47:20.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-18T11:47:20.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 123597 2024-09-18T11:47:20.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-18T11:47:20.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-18T11:47:20.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 1 2024-09-18T11:47:21.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-18T11:47:21.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 123597 2024-09-18T11:47:21.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-18T11:47:21.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-18T11:47:21.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0 2024-09-18T11:47:21.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 1 2024-09-18T11:47:21.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:293: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2024-09-18T11:47:21.202 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-09-18T11:47:21.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=123597 2024-09-18T11:47:21.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-09-18T11:47:21.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-18T11:47:21.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-09-18T11:47:21.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-18T11:47:21.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 123597 2024-09-18T11:47:21.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-18T11:47:21.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-18T11:47:21.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-09-18T11:47:21.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-18T11:47:21.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 123597 2024-09-18T11:47:21.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-18T11:47:21.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-18T11:47:21.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.2 2024-09-18T11:47:21.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-18T11:47:21.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 123597 2024-09-18T11:47:21.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-09-18T11:47:21.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-09-18T11:47:21.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-09-18T11:47:21.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: name_prefix=mgr 2024-09-18T11:47:21.508 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: find td/ceph-helpers 2024-09-18T11:47:21.508 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: grep 'mgr[^/]*\.pid' 2024-09-18T11:47:21.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-18T11:47:21.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2024-09-18T11:47:21.509 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2024-09-18T11:47:21.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=122644 2024-09-18T11:47:21.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-09-18T11:47:21.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-18T11:47:21.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-09-18T11:47:21.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-18T11:47:21.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 122644 2024-09-18T11:47:21.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-18T11:47:21.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-18T11:47:21.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-09-18T11:47:21.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-18T11:47:21.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 122644 2024-09-18T11:47:21.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-09-18T11:47:21.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-09-18T11:47:21.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-09-18T11:47:21.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: name_prefix=mon 2024-09-18T11:47:21.613 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: find td/ceph-helpers 2024-09-18T11:47:21.614 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: grep 'mon[^/]*\.pid' 2024-09-18T11:47:21.614 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-18T11:47:21.614 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:309: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2024-09-18T11:47:21.615 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mon.a.pid 2024-09-18T11:47:21.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=122558 2024-09-18T11:47:21.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-09-18T11:47:21.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-18T11:47:21.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-09-18T11:47:21.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-18T11:47:21.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 122558 2024-09-18T11:47:21.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-18T11:47:21.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-18T11:47:21.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-09-18T11:47:21.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-18T11:47:21.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 122558 2024-09-18T11:47:21.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-18T11:47:21.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-18T11:47:21.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.2 2024-09-18T11:47:21.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-18T11:47:21.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 122558 2024-09-18T11:47:21.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-09-18T11:47:21.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-09-18T11:47:21.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-09-18T11:47:21.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:310: test_kill_daemon: timeout 5 ceph status 2024-09-18T11:47:26.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:313: test_kill_daemon: teardown td/ceph-helpers 2024-09-18T11:47:26.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:47:26.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:47:26.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:47:26.923 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:47:26.923 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:47:26.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:47:26.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:47:26.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:47:26.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:47:26.928 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:47:26.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:47:26.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:47:26.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:47:26.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:47:26.931 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:47:26.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:47:26.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:47:26.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:47:26.933 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:47:26.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:47:26.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:47:26.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:47:26.941 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:47:26.941 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:47:26.941 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:47:26.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:47:26.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:47:26.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:47:26.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:47:26.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_kill_daemons td/ceph-helpers 2024-09-18T11:47:26.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:366: test_kill_daemons: local dir=td/ceph-helpers 2024-09-18T11:47:26.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:367: test_kill_daemons: setup td/ceph-helpers 2024-09-18T11:47:26.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:47:26.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:47:26.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:47:26.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:47:26.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:47:26.944 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:47:26.944 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:47:26.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:47:26.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:47:26.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:47:26.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:47:26.945 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:47:26.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:47:26.946 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:47:26.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:47:26.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:47:26.948 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:47:26.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:47:26.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:47:26.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:47:26.949 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:47:26.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:47:26.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:47:26.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:47:26.953 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:47:26.953 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:47:26.953 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:47:26.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:47:26.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:47:26.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:47:26.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:47:26.955 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:47:26.955 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:47:26.955 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:47:26.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:47:26.958 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:47:26.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:47:26.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:47:26.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:47:26.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:47:26.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:47:26.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:47:26.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:47:26.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:47:26.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:47:26.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:47:27.313 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:47:27.314 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:47:27.314 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:47:27.314 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:47:27.314 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:47:27.314 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:47:27.314 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:47:27.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:47:27.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:47:27.607 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:47:27.607 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:47:27.607 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:47:27.608 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:47:27.608 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:47:27.609 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:47:27.609 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:47:27.609 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:47:27.609 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:47:27.609 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:47:27.609 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:47:27.609 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:47:27.609 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:47:27.610 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:47:27.692 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:47:27.692 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:47:27.692 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:47:27.692 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:47:27.692 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:47:27.692 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:47:27.692 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:47:27.693 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:47:27.693 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:47:27.693 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:47:27.693 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:47:27.693 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:47:27.693 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:47:27.693 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:47:27.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:369: test_kill_daemons: run_mgr td/ceph-helpers x 2024-09-18T11:47:27.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:47:27.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:47:27.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:47:27.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:47:27.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:47:27.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:47:28.608 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:47:28.609 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:47:28.609 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:47:28.609 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:47:28.609 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:47:28.609 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:47:28.609 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:47:28.609 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:47:28.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:47:28.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:370: test_kill_daemons: run_osd td/ceph-helpers 0 2024-09-18T11:47:28.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:47:28.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:47:28.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:47:28.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:47:28.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:47:28.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:47:28.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:47:28.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:47:28.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:47:28.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:47:28.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:47:28.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:47:28.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:47:28.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:47:28.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:47:28.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:47:28.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:47:28.637 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:47:28.637 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:47:28.637 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:47:28.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:47:28.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:47:28.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:47:28.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:47:28.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:47:28.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:47:28.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:47:28.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:47:28.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:47:28.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:47:28.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:47:28.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:47:28.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:47:28.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:47:28.638 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:47:28.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=80e3b9bf-2baf-40f4-ba3b-709e9dbc0831 2024-09-18T11:47:28.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 80e3b9bf-2baf-40f4-ba3b-709e9dbc0831' 2024-09-18T11:47:28.639 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 80e3b9bf-2baf-40f4-ba3b-709e9dbc0831 2024-09-18T11:47:28.639 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:47:28.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDQvepm/NC+JhAAo717n1zliFTJ/FuXd3na5g== 2024-09-18T11:47:28.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDQvepm/NC+JhAAo717n1zliFTJ/FuXd3na5g=="}' 2024-09-18T11:47:28.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 80e3b9bf-2baf-40f4-ba3b-709e9dbc0831 -i td/ceph-helpers/0/new.json 2024-09-18T11:47:28.994 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:47:29.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:47:29.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDQvepm/NC+JhAAo717n1zliFTJ/FuXd3na5g== --osd-uuid 80e3b9bf-2baf-40f4-ba3b-709e9dbc0831 2024-09-18T11:47:29.029 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:47:29.026+0000 7f8818c08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:47:29.049 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:47:29.046+0000 7f8818c08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:47:29.050 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:47:29.048+0000 7f8818c08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:47:29.050 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:47:29.048+0000 7f8818c08440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:47:29.050 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:47:29.048+0000 7f8818c08440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:47:29.050 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:47:29.048+0000 7f8818c08440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:47:29.081 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:47:29.078+0000 7f8818c08440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:47:34.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:47:34.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:47:34.729 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:47:34.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:47:34.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:47:35.217 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:47:35.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:47:35.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:47:35.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:47:35.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:47:35.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:47:35.239 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:47:35.238+0000 7f5093e67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:47:35.241 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:47:35.240+0000 7f5093e67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:47:35.242 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:47:35.241+0000 7f5093e67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:47:35.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:47:35.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:47:35.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:47:35.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:47:35.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:47:35.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:35.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:47:35.567 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:47:35.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:35.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:35.759 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:47:35.758+0000 7f5093e67440 -1 Falling back to public interface 2024-09-18T11:47:35.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:36.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:36.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:36.925 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:47:36.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:47:36.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:36.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:37.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:38.281 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:47:38.279+0000 7f5093e67440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:47:38.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:38.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:38.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:47:38.284 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:47:38.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:38.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:38.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:39.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:39.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:39.655 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:47:39.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:47:39.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:39.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:40.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:41.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:41.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:41.143 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:47:41.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:47:41.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:41.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:41.535 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:42.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:42.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:42.537 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:47:42.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:47:42.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:42.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:42.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:43.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:43.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:43.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:47:43.980 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:47:43.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:43.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:44.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:45.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:45.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:45.378 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:47:45.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:47:45.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:45.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:45.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:46.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:46.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:46.780 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:47:46.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:47:46.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:46.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:47.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:48.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:48.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:48.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:47:48.310 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:47:48.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:48.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:48.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:49.706 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:49.706 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:49.707 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:47:49.707 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:47:49.707 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:49.707 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:50.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:51.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:51.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:51.087 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:47:51.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:47:51.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:51.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:52.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:53.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:53.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:53.023 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:47:53.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:47:53.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:53.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:53.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:54.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:54.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:54.812 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:47:54.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:47:54.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:54.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:55.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:56.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:56.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:56.527 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:47:56.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:47:56.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:56.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:57.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:47:58.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:47:58.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:47:58.329 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:47:58.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:47:58.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:47:58.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:47:59.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:00.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:00.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:00.337 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:48:00.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:48:00.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:00.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:01.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:02.026 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:02.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:02.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:48:02.027 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:48:02.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:02.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:02.410 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/125643,v1:127.0.0.1:6803/125643] [v2:127.0.0.1:6804/125643,v1:127.0.0.1:6805/125643] exists,up 80e3b9bf-2baf-40f4-ba3b-709e9dbc0831 2024-09-18T11:48:02.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:48:02.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:48:02.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:48:02.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:377: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2024-09-18T11:48:02.412 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:48:02.412 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:48:02.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:48:02.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:48:02.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:48:03.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 1 2024-09-18T11:48:03.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2024-09-18T11:48:03.418 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:48:03.418 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:48:03.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:48:03.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:48:03.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:48:03.724 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:48:03.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:385: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2024-09-18T11:48:03.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:48:03.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:48:03.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:48:03.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:48:03.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:48:03.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:48:03.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:389: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2024-09-18T11:48:03.832 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:48:03.832 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:48:03.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:48:03.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:48:03.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:48:04.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:48:04.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:390: test_kill_daemons: timeout 5 ceph status 2024-09-18T11:48:09.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:391: test_kill_daemons: teardown td/ceph-helpers 2024-09-18T11:48:09.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:48:09.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:48:09.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:48:09.145 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:48:09.145 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:48:09.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:48:09.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:48:09.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:48:09.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:48:09.150 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:48:09.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:48:09.151 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:48:09.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:48:09.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:48:09.153 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:48:09.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:48:09.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:48:09.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:48:09.154 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:48:09.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:48:09.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:48:09.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:48:09.162 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:48:09.162 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:48:09.162 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:48:09.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:48:09.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:48:09.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:48:09.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:48:09.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_objectstore_tool td/ceph-helpers 2024-09-18T11:48:09.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: local dir=td/ceph-helpers 2024-09-18T11:48:09.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: setup td/ceph-helpers 2024-09-18T11:48:09.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:48:09.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:48:09.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:48:09.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:48:09.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:48:09.165 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:48:09.165 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:48:09.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:48:09.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:48:09.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:48:09.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:48:09.166 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:48:09.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:48:09.167 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:48:09.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:48:09.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:48:09.169 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:48:09.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:48:09.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:48:09.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:48:09.170 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:48:09.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:48:09.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:48:09.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:48:09.173 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:48:09.173 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:48:09.173 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:48:09.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:48:09.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:48:09.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:48:09.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:48:09.176 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:48:09.176 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:48:09.176 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:48:09.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:48:09.178 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:48:09.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:48:09.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:48:09.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:48:09.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:48:09.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:48:09.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:48:09.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:48:09.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:48:09.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:48:09.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:48:09.545 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:48:09.545 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:48:09.545 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:48:09.545 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:48:09.545 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:48:09.545 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:48:09.545 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:48:09.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:48:09.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:48:09.814 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:48:09.814 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:48:09.814 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:48:09.814 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:48:09.815 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:48:09.815 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:48:09.815 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:48:09.815 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:48:09.815 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:48:09.815 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:48:09.816 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:48:09.816 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:48:09.816 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:48:09.816 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:48:09.898 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:48:09.898 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:48:09.898 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:48:09.898 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:48:09.898 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:48:09.898 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:48:09.898 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:48:09.898 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:48:09.899 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:48:09.899 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:48:09.899 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:48:09.899 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:48:09.899 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:48:09.899 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:48:09.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: local osd=0 2024-09-18T11:48:09.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: test_objectstore_tool: run_mgr td/ceph-helpers x 2024-09-18T11:48:09.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:48:09.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:48:09.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:48:09.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:48:09.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:48:09.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:48:10.817 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:48:10.817 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:48:10.817 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:48:10.817 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:48:10.817 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:48:10.817 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:48:10.817 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:48:10.818 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:48:10.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:48:10.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: test_objectstore_tool: run_osd td/ceph-helpers 0 2024-09-18T11:48:10.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:48:10.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:48:10.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:48:10.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:48:10.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:48:10.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:48:10.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:48:10.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:48:10.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:48:10.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:48:10.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:48:10.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:48:10.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:48:10.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:48:10.844 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:48:10.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:48:10.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:48:10.845 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:48:10.845 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:48:10.845 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:48:10.847 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:48:10.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:48:10.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:48:10.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:48:10.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:48:10.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:48:10.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:48:10.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:48:10.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:48:10.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:48:10.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:48:10.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:48:10.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:48:10.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:48:10.848 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:48:10.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0c4b91cb-8063-4de1-9361-712e346330dd 2024-09-18T11:48:10.849 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 0c4b91cb-8063-4de1-9361-712e346330dd 2024-09-18T11:48:10.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0c4b91cb-8063-4de1-9361-712e346330dd' 2024-09-18T11:48:10.849 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:48:10.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD6vepmYGskMxAAjvacmein8UTETRiJgfQojw== 2024-09-18T11:48:10.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD6vepmYGskMxAAjvacmein8UTETRiJgfQojw=="}' 2024-09-18T11:48:10.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0c4b91cb-8063-4de1-9361-712e346330dd -i td/ceph-helpers/0/new.json 2024-09-18T11:48:11.201 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:48:11.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:48:11.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD6vepmYGskMxAAjvacmein8UTETRiJgfQojw== --osd-uuid 0c4b91cb-8063-4de1-9361-712e346330dd 2024-09-18T11:48:11.236 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:11.234+0000 7ff24ea12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:48:11.247 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:11.245+0000 7ff24ea12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:48:11.248 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:11.246+0000 7ff24ea12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:48:11.248 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:11.247+0000 7ff24ea12440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:48:11.248 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:11.247+0000 7ff24ea12440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:48:11.248 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:11.247+0000 7ff24ea12440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:48:11.287 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:11.285+0000 7ff24ea12440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:48:16.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:48:16.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:48:16.625 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:48:16.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:48:16.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:48:17.119 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:48:17.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:48:17.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:48:17.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:48:17.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:48:17.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:48:17.142 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:17.140+0000 7f1dd8380440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:48:17.166 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:17.165+0000 7f1dd8380440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:48:17.167 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:17.166+0000 7f1dd8380440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:48:17.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:48:17.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:48:17.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:48:17.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:48:17.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:48:17.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:17.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:48:17.546 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:48:17.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:17.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:17.698 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:17.697+0000 7f1dd8380440 -1 Falling back to public interface 2024-09-18T11:48:17.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:18.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:18.909 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:48:18.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:18.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:48:18.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:18.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:19.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:20.201 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:20.200+0000 7f1dd8380440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:48:20.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:20.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:20.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:48:20.257 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:48:20.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:20.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:20.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:21.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:21.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:21.822 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:48:21.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:48:21.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:21.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:22.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:23.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:23.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:23.242 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:48:23.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:48:23.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:23.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:23.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:24.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:24.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:24.648 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:48:24.648 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:48:24.648 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:24.648 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:25.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:26.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:26.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:26.067 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:48:26.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:48:26.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:26.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:26.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:27.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:27.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:27.429 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:48:27.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:48:27.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:27.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:27.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:28.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:28.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:28.923 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:48:28.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:48:28.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:28.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:29.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:30.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:30.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:30.332 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:48:30.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:48:30.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:30.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:30.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:31.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:31.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:31.692 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:48:31.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:48:31.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:31.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:32.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:33.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:33.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:33.683 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:48:33.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:48:33.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:33.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:34.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:35.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:35.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:35.320 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:48:35.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:48:35.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:35.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:36.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:37.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:37.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:37.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:48:37.126 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:48:37.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:37.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:38.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:39.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:39.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:39.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:48:39.219 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:48:39.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:39.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:40.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:41.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:41.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:41.056 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:48:41.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:48:41.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:41.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:42.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:43.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:43.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:43.344 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:48:43.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:48:43.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:43.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:43.704 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/127716,v1:127.0.0.1:6803/127716] [v2:127.0.0.1:6804/127716,v1:127.0.0.1:6805/127716] exists,up 0c4b91cb-8063-4de1-9361-712e346330dd 2024-09-18T11:48:43.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:48:43.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:48:43.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:48:43.706 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: test_objectstore_tool: create_rbd_pool 2024-09-18T11:48:43.706 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:48:44.083 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:48:44.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:48:44.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:48:44.611 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:48:44.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:48:45.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:48:46.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: test_objectstore_tool: wait_for_clean 2024-09-18T11:48:46.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:48:46.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:48:46.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:48:46.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:48:46.595 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:48:46.595 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:48:46.595 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:48:46.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:48:46.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:48:46.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:48:46.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:48:46.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:48:46.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:48:46.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:48:46.691 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:48:47.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T11:48:47.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:48:47.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:48:47.041 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:48:47.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836484 2024-09-18T11:48:47.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836484 2024-09-18T11:48:47.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836484' 2024-09-18T11:48:47.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:48:47.218 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836484 2024-09-18T11:48:47.218 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:48:47.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:48:47.219 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836484 2024-09-18T11:48:47.219 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:48:47.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836484 2024-09-18T11:48:47.220 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836484 2024-09-18T11:48:47.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-09-18T11:48:47.221 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:48:47.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836482 -lt 21474836484 2024-09-18T11:48:47.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:48:48.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:48:48.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:48:48.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836484 -lt 21474836484 2024-09-18T11:48:48.923 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:48:48.924 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:48:48.924 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:48:49.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:48:49.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:48:49.398 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:48:49.398 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:48:49.399 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:48:49.399 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:48:49.399 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:48:49.399 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:48:49.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:48:49.749 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:48:49.749 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:48:49.749 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:48:50.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T11:48:50.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:48:50.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:48:50.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1316: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2024-09-18T11:48:50.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2024-09-18T11:48:50.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: test_objectstore_tool: diff - /etc/group 2024-09-18T11:48:50.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-09-18T11:48:50.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:48:50.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T11:48:50.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:48:50.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2024-09-18T11:48:50.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-09-18T11:48:50.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:48:50.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T11:48:50.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:48:50.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-09-18T11:48:50.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:48:50.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:48:50.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:48:50.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:48:50.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:48:50.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:48:50.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2024-09-18T11:48:50.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-09-18T11:48:50.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:48:50.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T11:48:50.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:48:50.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-09-18T11:48:50.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2024-09-18T11:48:52.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2024-09-18T11:48:52.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-18T11:48:52.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:48:52.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T11:48:52.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:48:52.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:48:52.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:48:52.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:48:52.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:48:52.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:48:52.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:48:52.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:48:52.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:48:52.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:48:52.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:48:52.213 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:48:52.213 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:48:52.213 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:48:52.214 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:48:52.214 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:48:52.214 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:48:52.214 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:48:52.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:48:52.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:48:52.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:48:52.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:48:52.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:48:52.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:48:52.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:48:52.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:48:52.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:48:52.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:48:52.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:48:52.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T11:48:52.215 INFO:tasks.workunit.client.0.smithi157.stderr:start osd.0 2024-09-18T11:48:52.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:48:52.216 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-18T11:48:52.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T11:48:52.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:48:52.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:48:52.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:48:52.237 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:52.235+0000 7fd77bf23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:48:52.239 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:52.238+0000 7fd77bf23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:48:52.240 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:52.239+0000 7fd77bf23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:48:52.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T11:48:52.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:48:52.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:48:52.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:48:52.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:48:52.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:52.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:48:52.610 INFO:tasks.workunit.client.0.smithi157.stderr:0 2024-09-18T11:48:52.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:52.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:52.758 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:52.757+0000 7fd77bf23440 -1 Falling back to public interface 2024-09-18T11:48:53.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:54.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:54.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:54.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:48:54.011 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-18T11:48:54.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:54.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:54.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:55.302 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:48:55.300+0000 7fd77bf23440 -1 osd.0 13 log_to_monitors true 2024-09-18T11:48:55.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:55.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:55.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:48:55.367 INFO:tasks.workunit.client.0.smithi157.stderr:2 2024-09-18T11:48:55.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:55.368 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:55.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:48:56.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:48:56.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:48:56.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:48:56.733 INFO:tasks.workunit.client.0.smithi157.stderr:3 2024-09-18T11:48:56.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:48:56.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:48:57.111 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 up in weight 1 up_from 16 up_thru 16 down_at 14 last_clean_interval [5,13) [v2:127.0.0.1:6802/129327,v1:127.0.0.1:6803/129327] [v2:127.0.0.1:6804/129327,v1:127.0.0.1:6805/129327] exists,up 0c4b91cb-8063-4de1-9361-712e346330dd 2024-09-18T11:48:57.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:48:57.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:48:57.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:48:57.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:48:57.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:48:57.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:48:57.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:48:57.112 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:48:57.112 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:48:57.112 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:48:57.112 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:48:57.112 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:48:57.112 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:48:57.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:48:57.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:48:57.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:48:57.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:48:57.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:48:57.194 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:48:57.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T11:48:57.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:48:57.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:48:57.587 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:48:57.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=68719476739 2024-09-18T11:48:57.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 68719476739 2024-09-18T11:48:57.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-68719476739' 2024-09-18T11:48:57.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:48:57.694 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-68719476739 2024-09-18T11:48:57.694 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:48:57.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:48:57.695 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-68719476739 2024-09-18T11:48:57.695 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:48:57.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=68719476739 2024-09-18T11:48:57.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 68719476739' 2024-09-18T11:48:57.696 INFO:tasks.workunit.client.0.smithi157.stderr:waiting osd.0 seq 68719476739 2024-09-18T11:48:57.696 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:48:58.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 68719476739 2024-09-18T11:48:58.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:48:59.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:48:59.068 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:48:59.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 68719476739 -lt 68719476739 2024-09-18T11:48:59.421 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:48:59.421 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:48:59.421 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:48:59.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:48:59.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:48:59.911 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:48:59.911 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:48:59.911 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:48:59.911 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:48:59.911 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:48:59.911 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:49:00.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:49:00.250 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:49:00.251 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:49:00.251 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:49:00.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T11:49:00.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:49:00.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:49:00.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1319: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-18T11:49:00.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-09-18T11:49:00.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:49:00.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T11:49:00.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:49:00.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-18T11:49:00.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-09-18T11:49:00.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:49:00.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T11:49:00.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:49:00.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-09-18T11:49:00.732 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:49:00.732 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:49:00.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:49:00.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:49:00.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:49:01.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:49:01.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-18T11:49:01.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-09-18T11:49:01.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:49:01.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T11:49:01.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:49:01.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-09-18T11:49:01.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2024-09-18T11:49:02.167 INFO:tasks.workunit.client.0.smithi157.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2024-09-18T11:49:02.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-09-18T11:49:02.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: return 1 2024-09-18T11:49:02.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: return 1 2024-09-18T11:49:02.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1320: test_objectstore_tool: teardown td/ceph-helpers 2024-09-18T11:49:02.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:49:02.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:49:02.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:49:02.433 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:49:02.433 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:49:02.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:49:02.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:49:02.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:49:02.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:49:02.540 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:49:02.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:49:02.541 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:49:02.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:49:02.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:49:02.542 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:49:02.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:49:02.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:49:02.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:49:02.544 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:49:02.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:49:02.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:49:02.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:49:02.553 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:49:02.553 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:49:02.553 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:49:02.553 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:49:02.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:49:02.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:49:02.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:49:02.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_pg_scrub td/ceph-helpers 2024-09-18T11:49:02.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1902: test_pg_scrub: local dir=td/ceph-helpers 2024-09-18T11:49:02.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1904: test_pg_scrub: setup td/ceph-helpers 2024-09-18T11:49:02.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:49:02.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:49:02.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:49:02.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:49:02.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:49:02.556 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:49:02.556 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:49:02.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:49:02.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:49:02.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:49:02.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:49:02.557 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:49:02.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:49:02.558 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:49:02.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:49:02.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:49:02.560 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:49:02.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:49:02.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:49:02.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:49:02.561 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:49:02.562 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:49:02.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:49:02.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:49:02.564 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:49:02.564 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:49:02.564 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:49:02.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:49:02.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:49:02.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:49:02.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:49:02.567 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:49:02.567 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:49:02.567 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:49:02.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:49:02.569 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:49:02.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:49:02.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:49:02.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:49:02.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1905: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:49:02.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:49:02.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:49:02.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:49:02.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:49:02.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:49:02.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:49:02.935 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:49:02.935 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:49:02.935 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:49:02.935 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:49:02.936 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:49:02.936 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:49:02.936 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:49:02.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:49:03.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:49:03.196 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:49:03.196 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:49:03.196 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:49:03.196 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:49:03.197 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:49:03.197 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:49:03.197 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:49:03.198 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:49:03.198 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:49:03.198 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:49:03.198 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:49:03.198 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:49:03.198 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:49:03.198 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:49:03.280 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:49:03.280 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:49:03.280 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:49:03.280 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:49:03.280 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:49:03.281 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:49:03.281 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:49:03.281 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:49:03.282 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:49:03.282 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:49:03.282 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:49:03.282 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:49:03.282 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:49:03.282 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:49:03.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: test_pg_scrub: run_mgr td/ceph-helpers x 2024-09-18T11:49:03.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:49:03.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:49:03.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:49:03.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:49:03.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:49:03.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:49:04.197 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:49:04.197 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:49:04.197 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:49:04.197 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:49:04.198 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:49:04.198 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:49:04.198 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:49:04.198 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:49:04.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:49:04.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: test_pg_scrub: run_osd td/ceph-helpers 0 2024-09-18T11:49:04.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:49:04.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:49:04.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:49:04.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:49:04.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:49:04.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:49:04.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:49:04.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:49:04.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:49:04.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:49:04.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:49:04.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:49:04.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:49:04.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:49:04.226 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:49:04.226 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:49:04.226 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:49:04.226 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:49:04.227 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:49:04.227 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:49:04.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:49:04.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:49:04.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:49:04.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:49:04.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:49:04.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:49:04.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:49:04.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:49:04.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:49:04.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:49:04.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:49:04.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:49:04.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:49:04.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:49:04.228 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:49:04.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=873f0dcc-273f-4a70-a451-512777192341 2024-09-18T11:49:04.228 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 873f0dcc-273f-4a70-a451-512777192341 2024-09-18T11:49:04.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 873f0dcc-273f-4a70-a451-512777192341' 2024-09-18T11:49:04.229 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:49:04.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAwvupmY+k/DhAAhFLHQLaaY5jDwXRUyyZLxg== 2024-09-18T11:49:04.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAwvupmY+k/DhAAhFLHQLaaY5jDwXRUyyZLxg=="}' 2024-09-18T11:49:04.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 873f0dcc-273f-4a70-a451-512777192341 -i td/ceph-helpers/0/new.json 2024-09-18T11:49:04.549 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:49:04.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:49:04.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAwvupmY+k/DhAAhFLHQLaaY5jDwXRUyyZLxg== --osd-uuid 873f0dcc-273f-4a70-a451-512777192341 2024-09-18T11:49:04.583 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:04.582+0000 7f8827f2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:49:04.596 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:04.594+0000 7f8827f2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:49:04.597 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:04.595+0000 7f8827f2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:49:04.597 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:04.596+0000 7f8827f2d440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:49:04.597 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:04.596+0000 7f8827f2d440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:49:04.597 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:04.596+0000 7f8827f2d440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:49:04.636 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:04.634+0000 7f8827f2d440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:49:09.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:49:09.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:49:09.937 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:49:09.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:49:09.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:49:10.416 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:49:10.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:49:10.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:49:10.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:49:10.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:49:10.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:49:10.439 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:10.437+0000 7fd868b49440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:49:10.441 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:10.440+0000 7fd868b49440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:49:10.442 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:10.441+0000 7fd868b49440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:49:10.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:49:10.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:49:10.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:49:10.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:49:10.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:49:10.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:10.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:49:10.768 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:49:10.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:10.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:10.971 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:10.970+0000 7fd868b49440 -1 Falling back to public interface 2024-09-18T11:49:11.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:49:12.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:49:12.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:12.123 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:49:12.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:49:12.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:12.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:12.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:49:13.479 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:13.477+0000 7fd868b49440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:49:13.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:49:13.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:13.485 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:49:13.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:49:13.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:13.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:13.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:49:14.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:49:14.862 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:49:14.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:14.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:49:14.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:14.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:15.339 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:49:16.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:49:16.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:16.341 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:49:16.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:49:16.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:16.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:16.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:49:17.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:49:17.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:17.721 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:49:17.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:49:17.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:17.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:18.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:49:19.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:49:19.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:19.154 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:49:19.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:49:19.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:19.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:19.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:49:20.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:49:20.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:20.549 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:49:20.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:49:20.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:20.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:20.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:49:21.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:49:21.948 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:49:21.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:21.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:49:21.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:21.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:22.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:49:23.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:49:23.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:23.441 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:49:23.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:49:23.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:23.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:23.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:49:24.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:49:24.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:24.833 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:49:24.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:49:24.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:24.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:25.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:49:26.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:49:26.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:26.746 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:49:26.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:49:26.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:26.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:27.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:49:28.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:49:28.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:28.399 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:49:28.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:49:28.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:28.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:29.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:49:30.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:49:30.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:30.163 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:49:30.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:49:30.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:30.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:31.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:49:32.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:49:32.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:32.314 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:49:32.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:49:32.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:32.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:33.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:49:34.365 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:49:34.365 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:34.365 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:49:34.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:49:34.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:34.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:35.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:49:36.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:49:36.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:36.353 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:49:36.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:49:36.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:36.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:36.713 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/131618,v1:127.0.0.1:6803/131618] [v2:127.0.0.1:6804/131618,v1:127.0.0.1:6805/131618] exists,up 873f0dcc-273f-4a70-a451-512777192341 2024-09-18T11:49:36.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:49:36.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:49:36.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:49:36.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: test_pg_scrub: create_rbd_pool 2024-09-18T11:49:36.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:49:37.089 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:49:37.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:49:37.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:49:37.569 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:49:37.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:49:38.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:49:39.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1909: test_pg_scrub: wait_for_clean 2024-09-18T11:49:39.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:49:39.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:49:39.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:49:39.602 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:49:39.603 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:49:39.603 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:49:39.603 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:49:39.603 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:49:39.603 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:49:39.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:49:39.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:49:39.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:49:39.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:49:39.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:49:39.699 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:49:40.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T11:49:40.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:49:40.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:49:40.062 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:49:40.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836487 2024-09-18T11:49:40.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836487 2024-09-18T11:49:40.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487' 2024-09-18T11:49:40.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:49:40.171 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836487 2024-09-18T11:49:40.171 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:49:40.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:49:40.172 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836487 2024-09-18T11:49:40.172 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:49:40.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836487 2024-09-18T11:49:40.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-18T11:49:40.174 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836487 2024-09-18T11:49:40.174 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:49:40.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-18T11:49:40.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:49:41.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:49:41.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:49:41.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-18T11:49:41.946 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:49:41.946 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:49:41.946 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:49:42.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:49:42.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:49:42.414 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:49:42.415 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:49:42.415 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:49:42.415 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:49:42.415 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:49:42.415 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:49:42.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:49:42.793 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:49:42.793 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:49:42.793 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:49:43.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T11:49:43.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:49:43.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:49:43.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1910: test_pg_scrub: pg_scrub 1.0 2024-09-18T11:49:43.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: pg_scrub: local pgid=1.0 2024-09-18T11:49:43.264 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: get_last_scrub_stamp 1.0 2024-09-18T11:49:43.264 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:49:43.265 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:49:43.265 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:49:43.265 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:49:43.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: local last_scrub=2024-09-18T11:49:37.394510+0000 2024-09-18T11:49:43.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_scrub: ceph pg scrub 1.0 2024-09-18T11:49:44.036 INFO:tasks.workunit.client.0.smithi157.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-18T11:49:44.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: wait_for_scrub 1.0 2024-09-18T11:49:37.394510+0000 2024-09-18T11:49:44.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T11:49:44.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T11:49:37.394510+0000 2024-09-18T11:49:44.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T11:49:44.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T11:49:44.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T11:49:44.047 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T11:49:44.047 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:49:44.047 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:49:44.047 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:49:44.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:49:44.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T11:49:37.394510+0000 '>' 2024-09-18T11:49:37.394510+0000 2024-09-18T11:49:44.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T11:49:45.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T11:49:45.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T11:49:45.394 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T11:49:45.394 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:49:45.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:49:45.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:49:45.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:49:45.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T11:49:37.394510+0000 '>' 2024-09-18T11:49:37.394510+0000 2024-09-18T11:49:45.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T11:49:46.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T11:49:46.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T11:49:46.749 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T11:49:46.749 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:49:46.749 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:49:46.749 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:49:46.749 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:49:47.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T11:49:37.394510+0000 '>' 2024-09-18T11:49:37.394510+0000 2024-09-18T11:49:47.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T11:49:48.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T11:49:48.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T11:49:48.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T11:49:48.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:49:48.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:49:48.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:49:48.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:49:48.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T11:49:44.522452+0000 '>' 2024-09-18T11:49:37.394510+0000 2024-09-18T11:49:48.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T11:49:48.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2024-09-18T11:49:48.460 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:49:48.460 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:49:48.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:49:48.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:49:48.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:49:48.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:49:48.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: test_pg_scrub: TIMEOUT=1 2024-09-18T11:49:48.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: test_pg_scrub: pg_scrub 1.0 2024-09-18T11:49:48.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: pg_scrub: local pgid=1.0 2024-09-18T11:49:48.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: get_last_scrub_stamp 1.0 2024-09-18T11:49:48.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:49:48.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:49:48.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:49:48.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:49:48.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: local last_scrub=2024-09-18T11:49:44.522452+0000 2024-09-18T11:49:48.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_scrub: ceph pg scrub 1.0 2024-09-18T11:49:49.234 INFO:tasks.workunit.client.0.smithi157.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-18T11:49:49.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: wait_for_scrub 1.0 2024-09-18T11:49:44.522452+0000 2024-09-18T11:49:49.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T11:49:49.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T11:49:44.522452+0000 2024-09-18T11:49:49.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T11:49:49.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T11:49:49.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 1 )) 2024-09-18T11:49:49.246 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T11:49:49.246 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:49:49.246 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:49:49.246 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:49:49.246 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:49:49.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T11:49:44.522452+0000 '>' 2024-09-18T11:49:44.522452+0000 2024-09-18T11:49:49.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T11:49:50.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T11:49:50.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 1 )) 2024-09-18T11:49:50.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: wait_for_scrub: return 1 2024-09-18T11:49:50.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1913: test_pg_scrub: teardown td/ceph-helpers 2024-09-18T11:49:50.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:49:50.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:49:50.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:49:50.587 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:49:50.587 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:49:50.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:49:50.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:49:50.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:49:50.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:49:50.694 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:49:50.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:49:50.696 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:49:50.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:49:50.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:49:50.697 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:49:50.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:49:50.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:49:50.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:49:50.699 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:49:50.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:49:50.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:49:50.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:49:50.708 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:49:50.708 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:49:50.708 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:49:50.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:49:50.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:49:50.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:49:50.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:49:50.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_repair td/ceph-helpers 2024-09-18T11:49:50.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1862: test_repair: local dir=td/ceph-helpers 2024-09-18T11:49:50.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1864: test_repair: setup td/ceph-helpers 2024-09-18T11:49:50.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:49:50.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:49:50.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:49:50.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:49:50.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:49:50.711 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:49:50.711 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:49:50.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:49:50.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:49:50.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:49:50.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:49:50.712 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:49:50.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:49:50.714 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:49:50.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:49:50.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:49:50.715 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:49:50.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:49:50.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:49:50.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:49:50.717 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:49:50.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:49:50.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:49:50.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:49:50.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:49:50.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:49:50.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:49:50.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:49:50.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:49:50.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:49:50.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:49:50.722 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:49:50.723 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:49:50.723 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:49:50.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:49:50.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:49:50.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:49:50.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:49:50.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:49:50.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1865: test_repair: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:49:50.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:49:50.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:49:50.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:49:50.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:49:50.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:49:50.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:49:51.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:49:51.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:49:51.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:49:51.094 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:49:51.094 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:49:51.094 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:49:51.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:49:51.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:49:51.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:49:51.354 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:49:51.354 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:49:51.354 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:49:51.354 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:49:51.355 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:49:51.355 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:49:51.356 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:49:51.356 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:49:51.356 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:49:51.356 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:49:51.356 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:49:51.356 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:49:51.356 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:49:51.356 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:49:51.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:49:51.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:49:51.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:49:51.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:49:51.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:49:51.438 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:49:51.438 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:49:51.439 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:49:51.439 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:49:51.439 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:49:51.439 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:49:51.440 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:49:51.440 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:49:51.440 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:49:51.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1866: test_repair: run_mgr td/ceph-helpers x 2024-09-18T11:49:51.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:49:51.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:49:51.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:49:51.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:49:51.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:49:51.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:49:52.356 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:49:52.356 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:49:52.356 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:49:52.356 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:49:52.356 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:49:52.356 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:49:52.356 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:49:52.357 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:49:52.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:49:52.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1867: test_repair: run_osd td/ceph-helpers 0 2024-09-18T11:49:52.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:49:52.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:49:52.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:49:52.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:49:52.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:49:52.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:49:52.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:49:52.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:49:52.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:49:52.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:49:52.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:49:52.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:49:52.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:49:52.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:49:52.387 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:49:52.387 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:49:52.387 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:49:52.387 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:49:52.387 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:49:52.388 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:49:52.388 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:49:52.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:49:52.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:49:52.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:49:52.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:49:52.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:49:52.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:49:52.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:49:52.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:49:52.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:49:52.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:49:52.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:49:52.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:49:52.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:49:52.389 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:49:52.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=adfbf605-f22e-4630-aa9c-62de8c7dbfa6 2024-09-18T11:49:52.390 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 adfbf605-f22e-4630-aa9c-62de8c7dbfa6 2024-09-18T11:49:52.390 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 adfbf605-f22e-4630-aa9c-62de8c7dbfa6' 2024-09-18T11:49:52.390 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:49:52.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBgvupmX/2zFxAAdEUw3yiUk0lMZILX0TJogQ== 2024-09-18T11:49:52.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBgvupmX/2zFxAAdEUw3yiUk0lMZILX0TJogQ=="}' 2024-09-18T11:49:52.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new adfbf605-f22e-4630-aa9c-62de8c7dbfa6 -i td/ceph-helpers/0/new.json 2024-09-18T11:49:52.698 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:49:52.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:49:52.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBgvupmX/2zFxAAdEUw3yiUk0lMZILX0TJogQ== --osd-uuid adfbf605-f22e-4630-aa9c-62de8c7dbfa6 2024-09-18T11:49:52.732 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:52.731+0000 7f0bfe069440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:49:52.745 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:52.744+0000 7f0bfe069440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:49:52.746 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:52.745+0000 7f0bfe069440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:49:52.747 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:52.745+0000 7f0bfe069440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:49:52.747 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:52.745+0000 7f0bfe069440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:49:52.747 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:52.745+0000 7f0bfe069440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:49:52.785 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:52.784+0000 7f0bfe069440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:49:58.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:49:58.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:49:58.367 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:49:58.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:49:58.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:49:58.848 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:49:58.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:49:58.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:49:58.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:49:58.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:49:58.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:49:58.871 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:58.869+0000 7f90b8b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:49:58.872 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:58.871+0000 7f90b8b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:49:58.874 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:58.872+0000 7f90b8b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:49:59.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:49:59.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:49:59.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:49:59.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:49:59.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:49:59.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:49:59.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:49:59.198 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:49:59.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:49:59.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:49:59.394 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:49:59.393+0000 7f90b8b23440 -1 Falling back to public interface 2024-09-18T11:49:59.548 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:00.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:00.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:00.550 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:50:00.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:50:00.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:00.551 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:00.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:01.894 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:50:01.892+0000 7f90b8b23440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:50:01.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:01.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:01.899 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:50:01.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:50:01.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:01.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:02.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:03.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:03.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:03.281 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:50:03.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:50:03.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:03.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:03.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:04.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:04.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:04.766 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:50:04.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:50:04.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:04.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:05.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:06.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:06.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:06.159 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:50:06.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:50:06.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:06.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:06.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:07.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:07.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:07.612 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:50:07.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:50:07.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:07.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:07.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:08.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:08.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:08.993 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:50:08.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:50:08.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:08.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:09.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:10.381 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:10.381 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:10.382 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:50:10.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:50:10.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:10.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:10.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:11.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:11.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:11.870 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:50:11.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:50:11.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:11.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:12.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:13.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:13.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:13.233 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:50:13.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:50:13.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:13.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:14.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:15.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:15.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:15.207 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:50:15.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:50:15.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:15.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:15.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:16.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:16.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:16.979 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:50:16.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:50:16.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:16.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:17.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:18.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:18.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:18.694 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:50:18.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:50:18.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:18.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:19.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:20.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:20.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:20.426 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:50:20.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:50:20.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:20.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:21.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:22.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:22.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:22.255 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:50:22.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:50:22.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:22.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:23.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:24.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:24.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:24.042 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:50:24.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:50:24.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:24.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:24.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:25.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:25.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:25.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:50:25.572 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:50:25.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:25.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:25.930 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/134139,v1:127.0.0.1:6803/134139] [v2:127.0.0.1:6804/134139,v1:127.0.0.1:6805/134139] exists,up adfbf605-f22e-4630-aa9c-62de8c7dbfa6 2024-09-18T11:50:25.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:50:25.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:50:25.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:50:25.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1868: test_repair: create_rbd_pool 2024-09-18T11:50:25.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:50:26.268 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:50:26.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:50:26.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:50:26.812 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:50:26.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:50:27.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:50:28.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1869: test_repair: wait_for_clean 2024-09-18T11:50:28.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:50:28.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:50:28.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:50:28.861 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:50:28.862 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:50:28.862 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:50:28.862 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:50:28.862 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:50:28.862 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:50:28.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:50:28.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:50:28.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:50:28.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:50:28.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:50:28.959 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:50:29.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T11:50:29.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:50:29.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:50:29.309 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:50:29.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836487 2024-09-18T11:50:29.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836487 2024-09-18T11:50:29.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487' 2024-09-18T11:50:29.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:50:29.418 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836487 2024-09-18T11:50:29.418 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:50:29.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:50:29.419 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836487 2024-09-18T11:50:29.419 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:50:29.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836487 2024-09-18T11:50:29.421 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836487 2024-09-18T11:50:29.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-18T11:50:29.421 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:50:29.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-18T11:50:29.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:50:30.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:50:30.782 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:50:31.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-18T11:50:31.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:50:31.144 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:50:31.144 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:50:31.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:50:31.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:50:31.619 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:50:31.619 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:50:31.619 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:50:31.619 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:50:31.619 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:50:31.619 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:50:31.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:50:31.966 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:50:31.967 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:50:31.967 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:50:32.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T11:50:32.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:50:32.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:50:32.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1870: test_repair: repair 1.0 2024-09-18T11:50:32.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-09-18T11:50:32.437 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-09-18T11:50:32.437 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:50:32.437 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:50:32.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:50:32.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:50:32.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T11:50:26.555009+0000 2024-09-18T11:50:32.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-09-18T11:50:33.153 INFO:tasks.workunit.client.0.smithi157.stderr:instructing pg 1.0 on osd.0 to repair 2024-09-18T11:50:33.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-09-18T11:50:26.555009+0000 2024-09-18T11:50:33.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T11:50:33.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T11:50:26.555009+0000 2024-09-18T11:50:33.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T11:50:33.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T11:50:33.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T11:50:33.165 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T11:50:33.165 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:50:33.165 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:50:33.165 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:50:33.165 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:50:33.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T11:50:26.555009+0000 '>' 2024-09-18T11:50:26.555009+0000 2024-09-18T11:50:33.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T11:50:34.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T11:50:34.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T11:50:34.506 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T11:50:34.507 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:50:34.507 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:50:34.507 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:50:34.507 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:50:34.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T11:50:26.555009+0000 '>' 2024-09-18T11:50:26.555009+0000 2024-09-18T11:50:34.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T11:50:35.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T11:50:35.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T11:50:35.862 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T11:50:35.862 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:50:35.862 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:50:35.862 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:50:35.862 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:50:36.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T11:50:26.555009+0000 '>' 2024-09-18T11:50:26.555009+0000 2024-09-18T11:50:36.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T11:50:37.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T11:50:37.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T11:50:37.219 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T11:50:37.219 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:50:37.220 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:50:37.220 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:50:37.220 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:50:37.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T11:50:26.555009+0000 '>' 2024-09-18T11:50:26.555009+0000 2024-09-18T11:50:37.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T11:50:38.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T11:50:38.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T11:50:38.574 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T11:50:38.574 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:50:38.574 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:50:38.574 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:50:38.574 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:50:38.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T11:50:26.555009+0000 '>' 2024-09-18T11:50:26.555009+0000 2024-09-18T11:50:38.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T11:50:39.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T11:50:39.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T11:50:39.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T11:50:39.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:50:39.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:50:39.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:50:39.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:50:40.280 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T11:50:26.555009+0000 '>' 2024-09-18T11:50:26.555009+0000 2024-09-18T11:50:40.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T11:50:41.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T11:50:41.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T11:50:41.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T11:50:41.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:50:41.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:50:41.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:50:41.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:50:41.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T11:50:26.555009+0000 '>' 2024-09-18T11:50:26.555009+0000 2024-09-18T11:50:41.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T11:50:42.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T11:50:42.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T11:50:42.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T11:50:42.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:50:42.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:50:42.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:50:42.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:50:42.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T11:50:36.231552+0000 '>' 2024-09-18T11:50:26.555009+0000 2024-09-18T11:50:42.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T11:50:42.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1871: test_repair: kill_daemons td/ceph-helpers KILL osd 2024-09-18T11:50:42.984 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:50:42.984 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:50:42.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:50:42.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:50:42.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:50:43.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:50:43.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1872: test_repair: TIMEOUT=1 2024-09-18T11:50:43.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1872: test_repair: repair 1.0 2024-09-18T11:50:43.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-09-18T11:50:43.089 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-09-18T11:50:43.089 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:50:43.090 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:50:43.090 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:50:43.090 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:50:43.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T11:50:36.231552+0000 2024-09-18T11:50:43.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-09-18T11:50:43.760 INFO:tasks.workunit.client.0.smithi157.stderr:instructing pg 1.0 on osd.0 to repair 2024-09-18T11:50:43.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-09-18T11:50:36.231552+0000 2024-09-18T11:50:43.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T11:50:43.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T11:50:36.231552+0000 2024-09-18T11:50:43.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T11:50:43.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T11:50:43.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 1 )) 2024-09-18T11:50:43.772 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T11:50:43.772 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T11:50:43.772 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T11:50:43.772 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T11:50:43.773 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T11:50:44.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T11:50:36.231552+0000 '>' 2024-09-18T11:50:36.231552+0000 2024-09-18T11:50:44.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T11:50:45.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T11:50:45.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 1 )) 2024-09-18T11:50:45.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: wait_for_scrub: return 1 2024-09-18T11:50:45.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1873: test_repair: teardown td/ceph-helpers 2024-09-18T11:50:45.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:50:45.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:50:45.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:50:45.114 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:50:45.114 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:50:45.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:50:45.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:50:45.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:50:45.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:50:45.221 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:50:45.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:50:45.222 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:50:45.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:50:45.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:50:45.224 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:50:45.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:50:45.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:50:45.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:50:45.225 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:50:45.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:50:45.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:50:45.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:50:45.234 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:50:45.234 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:50:45.234 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:50:45.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:50:45.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:50:45.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:50:45.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:50:45.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_run_in_background td/ceph-helpers 2024-09-18T11:50:45.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: test_run_in_background: local pids 2024-09-18T11:50:45.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: test_run_in_background: run_in_background pids sleep 1 2024-09-18T11:50:45.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T11:50:45.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T11:50:45.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 135711"' 2024-09-18T11:50:45.237 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 135711' 2024-09-18T11:50:45.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: test_run_in_background: run_in_background pids sleep 1 2024-09-18T11:50:45.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T11:50:45.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T11:50:45.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sleep 1 2024-09-18T11:50:45.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 135713"' 2024-09-18T11:50:45.238 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 135713' 2024-09-18T11:50:45.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/135714: /' 2024-09-18T11:50:45.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sleep 1 2024-09-18T11:50:45.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/135717: /' 2024-09-18T11:50:45.238 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: test_run_in_background: echo 135711 135713 2024-09-18T11:50:45.238 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: test_run_in_background: wc -w 2024-09-18T11:50:45.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: test_run_in_background: test 2 = 2 2024-09-18T11:50:45.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: test_run_in_background: wait 135711 135713 2024-09-18T11:50:46.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T11:50:46.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T11:50:46.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:50:46.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_run_mon td/ceph-helpers 2024-09-18T11:50:46.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:495: test_run_mon: local dir=td/ceph-helpers 2024-09-18T11:50:46.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:497: test_run_mon: setup td/ceph-helpers 2024-09-18T11:50:46.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:50:46.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:50:46.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:50:46.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:50:46.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:50:46.243 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:50:46.244 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:50:46.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:50:46.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:50:46.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:50:46.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:50:46.244 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:50:46.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:50:46.244 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:50:46.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:50:46.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:50:46.245 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:50:46.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:50:46.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:50:46.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:50:46.246 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:50:46.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:50:46.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:50:46.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:50:46.248 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:50:46.248 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:50:46.248 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:50:46.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:50:46.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:50:46.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:50:46.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:50:46.251 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:50:46.251 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:50:46.251 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:50:46.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:50:46.253 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:50:46.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:50:46.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:50:46.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:50:46.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:499: test_run_mon: run_mon td/ceph-helpers a 2024-09-18T11:50:46.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:50:46.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:50:46.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:50:46.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:50:46.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:50:46.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-18T11:50:46.613 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:50:46.613 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:50:46.613 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:50:46.614 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:50:46.614 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:50:46.614 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:50:46.614 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:50:46.614 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:50:46.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:50:46.874 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:50:46.874 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:50:46.874 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:50:46.874 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:50:46.874 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:50:46.875 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:50:46.875 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:50:46.875 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:50:46.875 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:50:46.876 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:50:46.876 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:50:46.876 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:50:46.876 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:50:46.876 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:50:46.959 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:50:46.959 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:50:46.959 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:50:46.959 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:50:46.960 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:50:46.960 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:50:46.960 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:50:46.960 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:50:46.960 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:50:46.960 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:50:46.960 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:50:46.960 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:50:46.961 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:50:46.961 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:50:47.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: ceph mon dump 2024-09-18T11:50:47.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: grep mon.a 2024-09-18T11:50:47.816 INFO:tasks.workunit.client.0.smithi157.stderr:dumped monmap epoch 1 2024-09-18T11:50:47.824 INFO:tasks.workunit.client.0.smithi157.stdout:0: v2:127.0.0.1:7109/0 mon.a 2024-09-18T11:50:47.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:501: test_run_mon: kill_daemons td/ceph-helpers 2024-09-18T11:50:47.825 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:50:47.826 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:50:47.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:50:47.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:50:47.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:50:49.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:50:49.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:503: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=3 2024-09-18T11:50:49.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:50:49.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:50:49.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:50:49.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:50:49.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:50:49.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=3 2024-09-18T11:50:49.155 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:50:49.153+0000 7f4016e3b900 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-09-18T11:50:49.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:50:49.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:50:49.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:50:49.158 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:50:49.158 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:50:49.158 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:50:49.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:50:49.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-18T11:50:49.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:50:49.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:50:49.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:50:49.414 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:50:49.414 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:50:49.414 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:50:49.415 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:50:49.415 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:50:49.415 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:50:49.415 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:50:49.415 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:50:49.415 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:50:49.416 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:50:49.416 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:50:49.416 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:50:49.496 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:50:49.496 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:50:49.496 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:50:49.496 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:50:49.496 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:50:49.496 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:50:49.496 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:50:49.496 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:50:49.496 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:50:49.497 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:50:49.497 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:50:49.497 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:50:49.497 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:50:49.497 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:50:49.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:504: test_run_mon: run_osd td/ceph-helpers 0 2024-09-18T11:50:49.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:50:49.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:50:49.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:50:49.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:50:49.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:50:49.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:50:49.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:50:49.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:50:49.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:50:49.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:50:49.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:50:49.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:50:49.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:50:49.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:50:49.575 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:50:49.575 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:50:49.575 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:50:49.575 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:50:49.576 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:50:49.576 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:50:49.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:50:49.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:50:49.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:50:49.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:50:49.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:50:49.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:50:49.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:50:49.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:50:49.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:50:49.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:50:49.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:50:49.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:50:49.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:50:49.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:50:49.577 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:50:49.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=498ea01e-7307-4101-a355-b6174ad79d20 2024-09-18T11:50:49.578 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 498ea01e-7307-4101-a355-b6174ad79d20 2024-09-18T11:50:49.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 498ea01e-7307-4101-a355-b6174ad79d20' 2024-09-18T11:50:49.578 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:50:49.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCZvupmCEsKIxAALC9+WAZMdM1VeFskLYykmQ== 2024-09-18T11:50:49.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCZvupmCEsKIxAALC9+WAZMdM1VeFskLYykmQ=="}' 2024-09-18T11:50:49.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 498ea01e-7307-4101-a355-b6174ad79d20 -i td/ceph-helpers/0/new.json 2024-09-18T11:50:49.870 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:50:49.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:50:49.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCZvupmCEsKIxAALC9+WAZMdM1VeFskLYykmQ== --osd-uuid 498ea01e-7307-4101-a355-b6174ad79d20 2024-09-18T11:50:49.904 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:50:49.902+0000 7f756d80e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:50:49.917 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:50:49.916+0000 7f756d80e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:50:49.918 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:50:49.917+0000 7f756d80e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:50:49.919 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:50:49.917+0000 7f756d80e440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:50:49.919 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:50:49.917+0000 7f756d80e440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:50:49.919 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:50:49.917+0000 7f756d80e440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:50:49.975 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:50:49.974+0000 7f756d80e440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:50:55.562 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:50:55.562 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:50:55.563 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:50:55.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:50:55.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:50:55.760 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:50:55.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:50:55.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:50:55.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:50:55.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:50:55.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:50:55.783 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:50:55.781+0000 7fe8c9c25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:50:55.801 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:50:55.800+0000 7fe8c9c25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:50:55.802 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:50:55.801+0000 7fe8c9c25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:50:55.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:50:55.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:50:55.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:50:55.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:50:55.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:50:55.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:55.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:50:55.973 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:50:55.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:55.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:56.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:56.332 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:50:56.331+0000 7fe8c9c25440 -1 Falling back to public interface 2024-09-18T11:50:57.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:57.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:57.177 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:50:57.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:50:57.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:57.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:57.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:58.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:58.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:58.364 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:50:58.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:50:58.365 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:58.365 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:58.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:50:58.902 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:50:58.901+0000 7fe8c9c25440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:50:59.548 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:50:59.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:50:59.549 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:50:59.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:50:59.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:50:59.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:50:59.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:00.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:00.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:00.908 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:51:00.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:51:00.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:00.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:01.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:02.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:02.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:02.096 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:51:02.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:51:02.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:02.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:02.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:03.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:03.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:03.285 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:51:03.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:51:03.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:03.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:03.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:04.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:04.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:04.477 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:51:04.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:51:04.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:04.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:04.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:05.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:05.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:05.666 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:51:05.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:51:05.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:05.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:05.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:06.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:06.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:06.855 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:51:06.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:51:06.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:06.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:07.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:08.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:08.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:08.045 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:51:08.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:51:08.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:08.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:08.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:09.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:09.234 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:51:09.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:09.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:51:09.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:09.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:09.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:10.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:10.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:10.423 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:51:10.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:51:10.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:10.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:10.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:11.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:11.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:11.608 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:51:11.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:51:11.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:11.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:11.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:12.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:12.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:12.797 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:51:12.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:51:12.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:12.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:12.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:13.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:13.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:13.981 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:51:13.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:51:13.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:13.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:14.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:15.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:15.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:15.169 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:51:15.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:51:15.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:15.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:15.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:16.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:16.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:16.355 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:51:16.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:51:16.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:16.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:16.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:17.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:17.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:17.545 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T11:51:17.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:51:17.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:17.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:17.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:18.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:18.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:18.735 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-18T11:51:18.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T11:51:18.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:18.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:18.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:19.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:19.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:19.925 INFO:tasks.workunit.client.0.smithi157.stdout:20 2024-09-18T11:51:19.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T11:51:19.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:19.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:51:20.162 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/136746,v1:127.0.0.1:6801/136746] [v2:127.0.0.1:6802/136746,v1:127.0.0.1:6803/136746] exists,up 498ea01e-7307-4101-a355-b6174ad79d20 2024-09-18T11:51:20.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:51:20.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:51:20.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:51:20.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:505: test_run_mon: run_osd td/ceph-helpers 1 2024-09-18T11:51:20.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:51:20.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:51:20.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:51:20.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:51:20.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-18T11:51:20.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:51:20.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:51:20.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:51:20.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:51:20.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-18T11:51:20.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-18T11:51:20.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:51:20.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:51:20.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:51:20.166 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:51:20.166 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:51:20.166 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:51:20.166 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:51:20.167 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:51:20.167 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:51:20.167 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:51:20.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:51:20.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:51:20.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:51:20.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:51:20.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:51:20.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:51:20.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:51:20.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:51:20.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:51:20.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:51:20.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:51:20.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:51:20.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-18T11:51:20.168 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:51:20.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4282b53c-4da5-4ca3-9c8e-38d13eb83fda 2024-09-18T11:51:20.169 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 4282b53c-4da5-4ca3-9c8e-38d13eb83fda 2024-09-18T11:51:20.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4282b53c-4da5-4ca3-9c8e-38d13eb83fda' 2024-09-18T11:51:20.169 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:51:20.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC4vupmu5iyChAAZ1pywBtd1vf2b40kOaxh+A== 2024-09-18T11:51:20.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC4vupmu5iyChAAZ1pywBtd1vf2b40kOaxh+A=="}' 2024-09-18T11:51:20.181 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4282b53c-4da5-4ca3-9c8e-38d13eb83fda -i td/ceph-helpers/1/new.json 2024-09-18T11:51:20.458 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:51:20.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-18T11:51:20.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4vupmu5iyChAAZ1pywBtd1vf2b40kOaxh+A== --osd-uuid 4282b53c-4da5-4ca3-9c8e-38d13eb83fda 2024-09-18T11:51:20.492 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:20.491+0000 7f4442d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:51:20.494 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:20.492+0000 7f4442d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:51:20.495 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:20.493+0000 7f4442d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:51:20.495 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:20.494+0000 7f4442d23440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:51:20.495 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:20.494+0000 7f4442d23440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:51:20.495 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:20.494+0000 7f4442d23440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:51:20.566 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:20.565+0000 7f4442d23440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-18T11:51:26.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-18T11:51:26.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:51:26.386 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T11:51:26.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:51:26.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:51:26.584 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T11:51:26.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:51:26.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:51:26.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:51:26.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:51:26.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:51:26.606 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:26.605+0000 7f913d921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:51:26.620 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:26.618+0000 7f913d921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:51:26.621 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:26.619+0000 7f913d921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:51:26.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:51:26.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:51:26.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:51:26.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:51:26.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:51:26.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:26.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:51:26.792 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:51:26.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:26.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:26.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:27.156 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:27.155+0000 7f913d921440 -1 Falling back to public interface 2024-09-18T11:51:27.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:27.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:27.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:51:27.992 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:51:27.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:27.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:28.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:29.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:29.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:29.179 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:51:29.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:51:29.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:29.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:29.368 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:29.738 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:29.736+0000 7f913d921440 -1 osd.1 0 log_to_monitors true 2024-09-18T11:51:30.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:30.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:30.370 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:51:30.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:51:30.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:30.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:30.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:31.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:31.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:31.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:51:31.744 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:51:31.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:31.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:31.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:32.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:32.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:32.933 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:51:32.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:51:32.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:32.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:33.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:34.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:34.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:34.119 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:51:34.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:51:34.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:34.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:34.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:35.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:35.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:35.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:51:35.310 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:51:35.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:35.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:35.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:36.499 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:36.499 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:51:36.499 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:36.499 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:51:36.500 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:36.500 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:36.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:37.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:37.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:37.688 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:51:37.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:51:37.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:37.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:37.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:38.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:38.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:38.877 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:51:38.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:51:38.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:38.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:39.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:40.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:40.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:40.065 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:51:40.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:51:40.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:40.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:40.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:41.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:41.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:41.253 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:51:41.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:51:41.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:41.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:41.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:42.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:42.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:42.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:51:42.442 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:51:42.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:42.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:42.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:43.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:43.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:43.629 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:51:43.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:51:43.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:43.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:43.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:44.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:44.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:44.820 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:51:44.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:51:44.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:44.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:45.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:46.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:46.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:46.008 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:51:46.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:51:46.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:46.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:46.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:47.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:47.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:47.200 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:51:47.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:51:47.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:47.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:47.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:48.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:48.388 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T11:51:48.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:48.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:51:48.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:48.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:48.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:49.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:49.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:49.578 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-18T11:51:49.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T11:51:49.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:49.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:49.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:50.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:50.767 INFO:tasks.workunit.client.0.smithi157.stdout:20 2024-09-18T11:51:50.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:50.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T11:51:50.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:50.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:50.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:51.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:51.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:51.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-18T11:51:51.956 INFO:tasks.workunit.client.0.smithi157.stdout:21 2024-09-18T11:51:51.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:51.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:51:52.136 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/138576,v1:127.0.0.1:6809/138576] [v2:127.0.0.1:6810/138576,v1:127.0.0.1:6811/138576] exists,up 4282b53c-4da5-4ca3-9c8e-38d13eb83fda 2024-09-18T11:51:52.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:51:52.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:51:52.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:51:52.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:506: test_run_mon: run_osd td/ceph-helpers 2 2024-09-18T11:51:52.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:51:52.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:51:52.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:51:52.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:51:52.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-09-18T11:51:52.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:51:52.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:51:52.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:51:52.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:51:52.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-09-18T11:51:52.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-09-18T11:51:52.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:51:52.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:51:52.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:51:52.141 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:51:52.141 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:51:52.141 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:51:52.142 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:51:52.142 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:51:52.142 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:51:52.142 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:51:52.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:51:52.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:51:52.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:51:52.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:51:52.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:51:52.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:51:52.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:51:52.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:51:52.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:51:52.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:51:52.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:51:52.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:51:52.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-09-18T11:51:52.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:51:52.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6ccb5b16-18ae-4d15-b815-a8a120f5dc8a 2024-09-18T11:51:52.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6ccb5b16-18ae-4d15-b815-a8a120f5dc8a' 2024-09-18T11:51:52.143 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 6ccb5b16-18ae-4d15-b815-a8a120f5dc8a 2024-09-18T11:51:52.144 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:51:52.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDYvupmzogpCRAAFVnHJoGtWD4uCDfcMt7K/w== 2024-09-18T11:51:52.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDYvupmzogpCRAAFVnHJoGtWD4uCDfcMt7K/w=="}' 2024-09-18T11:51:52.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6ccb5b16-18ae-4d15-b815-a8a120f5dc8a -i td/ceph-helpers/2/new.json 2024-09-18T11:51:52.457 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:51:52.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-09-18T11:51:52.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDYvupmzogpCRAAFVnHJoGtWD4uCDfcMt7K/w== --osd-uuid 6ccb5b16-18ae-4d15-b815-a8a120f5dc8a 2024-09-18T11:51:52.491 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:52.489+0000 7f2567c39440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:51:52.493 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:52.491+0000 7f2567c39440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:51:52.494 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:52.492+0000 7f2567c39440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:51:52.494 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:52.493+0000 7f2567c39440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-18T11:51:52.494 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:52.493+0000 7f2567c39440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-18T11:51:52.494 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:52.493+0000 7f2567c39440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-18T11:51:52.574 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:52.573+0000 7f2567c39440 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-09-18T11:51:58.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-09-18T11:51:58.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:51:58.160 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-18T11:51:58.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:51:58.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:51:58.357 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-18T11:51:58.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:51:58.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:51:58.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:51:58.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:51:58.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:51:58.379 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:58.378+0000 7fbe1491d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:51:58.393 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:58.391+0000 7fbe1491d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:51:58.394 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:58.392+0000 7fbe1491d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:51:58.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:51:58.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:51:58.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:51:58.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:51:58.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:51:58.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:58.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:51:58.566 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:51:58.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:58.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:51:58.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:51:58.926 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:51:58.925+0000 7fbe1491d440 -1 Falling back to public interface 2024-09-18T11:51:59.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:51:59.775 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:51:59.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:51:59.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:51:59.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:51:59.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:51:59.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:00.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:00.963 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:52:00.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:00.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:52:00.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:00.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:01.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:01.488 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:01.486+0000 7fbe1491d440 -1 osd.2 0 log_to_monitors true 2024-09-18T11:52:02.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:02.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:02.144 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:52:02.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:52:02.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:02.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:02.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:03.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:03.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:03.543 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:52:03.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:52:03.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:03.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:03.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:04.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:04.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:04.732 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:52:04.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:52:04.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:04.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:04.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:05.920 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:05.920 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:05.921 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:52:05.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:52:05.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:05.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:06.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:07.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:07.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:07.111 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:52:07.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:52:07.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:07.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:07.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:08.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:08.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:08.299 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:52:08.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:52:08.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:08.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:08.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:09.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:09.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:09.489 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:52:09.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:52:09.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:09.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:09.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:10.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:10.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:10.679 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:52:10.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:52:10.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:10.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:10.865 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:11.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:11.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:11.867 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:52:11.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:52:11.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:11.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:12.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:13.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:13.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:13.058 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:52:13.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:52:13.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:13.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:13.247 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:14.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:14.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:14.249 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:52:14.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:52:14.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:14.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:14.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:15.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:15.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:15.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:52:15.437 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:52:15.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:15.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:15.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:16.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:16.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:16.626 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:52:16.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:52:16.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:16.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:16.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:17.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:17.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:17.811 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:52:17.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:52:17.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:17.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:17.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:18.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:18.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:18.996 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:52:18.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:52:18.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:18.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:19.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:20.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:20.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:20.184 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T11:52:20.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:52:20.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:20.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:20.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:21.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:21.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:21.374 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-18T11:52:21.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T11:52:21.375 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:21.375 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:21.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:22.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:22.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:22.561 INFO:tasks.workunit.client.0.smithi157.stdout:20 2024-09-18T11:52:22.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T11:52:22.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:22.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:22.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:23.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:23.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:23.756 INFO:tasks.workunit.client.0.smithi157.stdout:21 2024-09-18T11:52:23.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-18T11:52:23.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:23.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:52:23.946 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/140430,v1:127.0.0.1:6817/140430] [v2:127.0.0.1:6818/140430,v1:127.0.0.1:6819/140430] exists,up 6ccb5b16-18ae-4d15-b815-a8a120f5dc8a 2024-09-18T11:52:23.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:52:23.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:52:23.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:52:23.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:507: test_run_mon: create_rbd_pool 2024-09-18T11:52:23.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:52:24.185 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:52:24.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:52:24.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:52:24.552 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:52:24.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:52:25.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:52:27.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: ceph osd dump 2024-09-18T11:52:27.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: grep 'pool 1 '\''rbd'\''' 2024-09-18T11:52:27.590 INFO:tasks.workunit.client.0.smithi157.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 23 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd 2024-09-18T11:52:27.591 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: get_asok_path mon.a 2024-09-18T11:52:27.591 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:52:27.591 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:52:27.591 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:52:27.592 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:27.592 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:27.592 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:52:27.592 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: CEPH_ARGS= 2024-09-18T11:52:27.592 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get osd_pool_default_size 2024-09-18T11:52:27.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: local 'size={"osd_pool_default_size":"3"}' 2024-09-18T11:52:27.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:511: test_run_mon: test '{"osd_pool_default_size":"3"}' = '{"osd_pool_default_size":"3"}' 2024-09-18T11:52:27.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: CEPH_ARGS= 2024-09-18T11:52:27.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: ceph status 2024-09-18T11:52:27.731 INFO:tasks.workunit.client.0.smithi157.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-18T11:52:27.731 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:27.729+0000 7f0fddcc9640 -1 failed for service _ceph-mon._tcp 2024-09-18T11:52:27.731 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:27.729+0000 7f0fddcc9640 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-18T11:52:27.731 INFO:tasks.workunit.client.0.smithi157.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2024-09-18T11:52:27.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: CEPH_ARGS= 2024-09-18T11:52:27.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: ceph --conf td/ceph-helpers/ceph.conf status 2024-09-18T11:52:27.799 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:27.797+0000 7f537c6dd640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-18T11:52:27.799 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:27.797+0000 7f537c6dd640 -1 AuthRegistry(0x7f5374064520) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-18T11:52:27.799 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:27.798+0000 7f537c6dd640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-18T11:52:27.799 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:27.798+0000 7f537c6dd640 -1 AuthRegistry(0x7f537c6dbff0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-18T11:52:27.801 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:27.799+0000 7f537c6dd640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-18T11:52:27.801 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:27.799+0000 7f537c6dd640 -1 AuthRegistry(0x7f5374061630) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-18T11:52:27.801 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:27.799+0000 7f537c6dd640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-18T11:52:27.801 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:27.799+0000 7f537c6dd640 -1 AuthRegistry(0x7f53740f6cf0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-18T11:52:27.801 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:27.800+0000 7f537c6dd640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-18T11:52:27.801 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:27.800+0000 7f537c6dd640 -1 AuthRegistry(0x7f5374061630) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-18T11:52:27.899 INFO:tasks.workunit.client.0.smithi157.stdout: cluster: 2024-09-18T11:52:27.899 INFO:tasks.workunit.client.0.smithi157.stdout: id: 92255036-ee37-4797-aa7f-f8e6b3f0325b 2024-09-18T11:52:27.899 INFO:tasks.workunit.client.0.smithi157.stdout: health: HEALTH_OK 2024-09-18T11:52:27.899 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-18T11:52:27.899 INFO:tasks.workunit.client.0.smithi157.stdout: services: 2024-09-18T11:52:27.899 INFO:tasks.workunit.client.0.smithi157.stdout: mon: 1 daemons, quorum a (age 98s) 2024-09-18T11:52:27.899 INFO:tasks.workunit.client.0.smithi157.stdout: mgr: no daemons active 2024-09-18T11:52:27.899 INFO:tasks.workunit.client.0.smithi157.stdout: osd: 3 osds: 3 up (since 4s), 3 in (since 35s) 2024-09-18T11:52:27.899 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-18T11:52:27.899 INFO:tasks.workunit.client.0.smithi157.stdout: data: 2024-09-18T11:52:27.899 INFO:tasks.workunit.client.0.smithi157.stdout: pools: 0 pools, 0 pgs 2024-09-18T11:52:27.900 INFO:tasks.workunit.client.0.smithi157.stdout: objects: 0 objects, 0 B 2024-09-18T11:52:27.900 INFO:tasks.workunit.client.0.smithi157.stdout: usage: 0 B used, 0 B / 0 B avail 2024-09-18T11:52:27.900 INFO:tasks.workunit.client.0.smithi157.stdout: pgs: 2024-09-18T11:52:27.900 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-18T11:52:27.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:516: test_run_mon: kill_daemons td/ceph-helpers 2024-09-18T11:52:27.908 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:52:27.908 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:52:27.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:52:27.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:52:27.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:52:33.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:52:33.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:518: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:52:33.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:52:33.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:52:33.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:52:33.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:52:33.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:52:33.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:52:33.242 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:33.240+0000 7f0de5567900 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-09-18T11:52:33.243 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:52:33.244 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:52:33.244 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:52:33.244 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:52:33.244 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:33.245 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:33.245 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:52:33.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:52:33.518 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:52:33.518 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:52:33.518 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:52:33.518 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:52:33.519 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:52:33.520 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:52:33.520 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:52:33.520 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:52:33.520 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:52:33.520 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:52:33.521 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:33.521 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:33.521 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:52:33.521 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:52:33.521 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:52:33.603 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:52:33.603 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:52:33.603 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:52:33.604 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:52:33.604 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:52:33.604 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:52:33.604 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:52:33.604 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:52:33.604 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:52:33.604 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:33.604 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:33.604 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:52:33.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:52:33.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:52:33.680 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: get_asok_path mon.a 2024-09-18T11:52:33.680 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:52:33.680 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:52:33.681 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:52:33.681 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:33.681 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:33.681 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:52:33.681 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: CEPH_ARGS= 2024-09-18T11:52:33.681 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get osd_pool_default_size 2024-09-18T11:52:33.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: local 'size={"osd_pool_default_size":"1"}' 2024-09-18T11:52:33.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:521: test_run_mon: test '{"osd_pool_default_size":"1"}' = '{"osd_pool_default_size":"1"}' 2024-09-18T11:52:33.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:522: test_run_mon: kill_daemons td/ceph-helpers 2024-09-18T11:52:33.754 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:52:33.754 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:52:33.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:52:33.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:52:33.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:52:33.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:52:33.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: CEPH_ARGS='--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd_pool_default_size=2' 2024-09-18T11:52:33.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: run_mon td/ceph-helpers a 2024-09-18T11:52:33.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:52:33.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:52:33.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:52:33.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:52:33.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:52:33.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-18T11:52:33.877 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:33.876+0000 7f9d7ec2d900 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-09-18T11:52:33.879 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:52:33.879 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:52:33.879 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:52:33.880 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:52:33.880 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:33.880 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:33.880 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:52:33.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:52:34.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:52:34.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:52:34.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:52:34.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:52:34.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:52:34.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:52:34.158 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:52:34.158 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:52:34.158 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:52:34.158 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:52:34.158 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:34.158 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:34.159 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:52:34.159 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:52:34.159 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:52:34.231 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:52:34.231 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:52:34.232 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:52:34.232 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:52:34.232 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:52:34.232 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:52:34.232 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:52:34.232 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:52:34.232 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:52:34.232 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:34.232 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:34.232 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:52:34.232 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:52:34.232 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:52:34.306 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: get_asok_path mon.a 2024-09-18T11:52:34.307 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:52:34.307 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:52:34.307 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:52:34.307 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:34.307 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:34.307 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:52:34.307 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: CEPH_ARGS= 2024-09-18T11:52:34.307 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get osd_pool_default_size 2024-09-18T11:52:34.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: local 'size={"osd_pool_default_size":"2"}' 2024-09-18T11:52:34.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: test_run_mon: test '{"osd_pool_default_size":"2"}' = '{"osd_pool_default_size":"2"}' 2024-09-18T11:52:34.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:529: test_run_mon: kill_daemons td/ceph-helpers 2024-09-18T11:52:34.380 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:52:34.380 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:52:34.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:52:34.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:52:34.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:52:34.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:52:34.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:531: test_run_mon: teardown td/ceph-helpers 2024-09-18T11:52:34.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:52:34.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:52:34.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:52:34.687 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:52:34.687 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:52:34.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:52:34.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:52:34.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:52:34.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:52:34.691 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:52:34.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:52:34.692 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:52:34.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:52:34.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:52:34.698 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:52:34.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:52:34.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:52:34.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:52:34.698 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:52:34.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:52:34.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:52:34.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:52:34.707 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:52:34.708 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:34.708 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:34.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:52:34.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:52:34.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:52:34.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:52:34.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_run_osd td/ceph-helpers 2024-09-18T11:52:34.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:745: test_run_osd: local dir=td/ceph-helpers 2024-09-18T11:52:34.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:747: test_run_osd: setup td/ceph-helpers 2024-09-18T11:52:34.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:52:34.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:52:34.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:52:34.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:52:34.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:52:34.710 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:52:34.711 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:52:34.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:52:34.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:52:34.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:52:34.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:52:34.711 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:52:34.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:52:34.712 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:52:34.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:52:34.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:52:34.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:52:34.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:52:34.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:52:34.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:52:34.715 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:52:34.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:52:34.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:52:34.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:52:34.717 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:52:34.717 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:34.717 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:34.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:52:34.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:52:34.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:52:34.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:52:34.719 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:52:34.719 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:34.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:34.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:52:34.721 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:52:34.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:52:34.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:52:34.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:52:34.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:749: test_run_osd: run_mon td/ceph-helpers a 2024-09-18T11:52:34.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:52:34.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:52:34.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:52:34.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:52:34.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:52:34.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-18T11:52:35.101 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:52:35.102 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:52:35.102 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:52:35.102 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:52:35.102 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:35.102 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:35.102 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:52:35.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:52:35.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:52:35.370 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:52:35.370 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:52:35.370 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:52:35.370 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:52:35.370 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:52:35.370 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:52:35.370 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:52:35.370 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:52:35.371 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:52:35.371 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:35.371 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:35.372 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:52:35.372 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:52:35.372 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:52:35.446 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:52:35.447 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:52:35.447 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:52:35.447 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:52:35.447 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:52:35.447 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:52:35.447 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:52:35.447 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:52:35.447 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:52:35.447 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:35.447 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:35.448 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:52:35.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:52:35.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:52:35.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:750: test_run_osd: run_mgr td/ceph-helpers x 2024-09-18T11:52:35.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:52:35.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:52:35.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:52:35.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:52:35.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:52:35.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:52:36.381 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:52:36.381 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:52:36.381 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:52:36.381 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:52:36.381 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:36.381 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:36.381 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:52:36.381 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:52:36.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:52:36.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:752: test_run_osd: run_osd td/ceph-helpers 0 2024-09-18T11:52:36.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:52:36.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:52:36.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:52:36.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:52:36.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:52:36.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:52:36.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:52:36.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:52:36.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:52:36.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:52:36.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:52:36.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:52:36.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:52:36.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:52:36.410 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:52:36.410 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:52:36.410 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:52:36.410 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:52:36.410 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:52:36.410 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:52:36.410 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:52:36.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:52:36.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:52:36.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:52:36.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:52:36.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:52:36.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:52:36.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:52:36.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:52:36.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:52:36.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:52:36.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:52:36.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:52:36.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:52:36.412 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:52:36.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4cadedb9-d5a3-4dd3-b619-bdd5357aba48 2024-09-18T11:52:36.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4cadedb9-d5a3-4dd3-b619-bdd5357aba48' 2024-09-18T11:52:36.412 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 4cadedb9-d5a3-4dd3-b619-bdd5357aba48 2024-09-18T11:52:36.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:52:36.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAEv+pmdHZAGRAAGvcvNvqpBJjeIBXWANe5VA== 2024-09-18T11:52:36.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAEv+pmdHZAGRAAGvcvNvqpBJjeIBXWANe5VA=="}' 2024-09-18T11:52:36.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4cadedb9-d5a3-4dd3-b619-bdd5357aba48 -i td/ceph-helpers/0/new.json 2024-09-18T11:52:36.766 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:52:36.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:52:36.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAEv+pmdHZAGRAAGvcvNvqpBJjeIBXWANe5VA== --osd-uuid 4cadedb9-d5a3-4dd3-b619-bdd5357aba48 2024-09-18T11:52:36.801 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:36.799+0000 7f61f4f24440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:52:36.829 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:36.828+0000 7f61f4f24440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:52:36.830 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:36.829+0000 7f61f4f24440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:52:36.831 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:36.829+0000 7f61f4f24440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:52:36.831 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:36.829+0000 7f61f4f24440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:52:36.831 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:36.829+0000 7f61f4f24440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:52:36.869 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:36.868+0000 7f61f4f24440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:52:42.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:52:42.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:52:42.517 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:52:42.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:52:42.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:52:43.003 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:52:43.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:52:43.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:52:43.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:52:43.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:52:43.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:52:43.026 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:43.025+0000 7f6cf8e3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:52:43.028 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:43.027+0000 7f6cf8e3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:52:43.029 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:43.028+0000 7f6cf8e3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:52:43.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:52:43.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:52:43.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:52:43.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:52:43.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:52:43.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:43.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:52:43.356 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:52:43.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:43.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:52:43.547 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:43.546+0000 7f6cf8e3b440 -1 Falling back to public interface 2024-09-18T11:52:43.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:44.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:44.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:44.725 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:52:44.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:52:44.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:44.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:52:45.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:46.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:46.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:46.082 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:52:46.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:52:46.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:46.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:52:46.102 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:52:46.101+0000 7f6cf8e3b440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:52:46.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:47.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:47.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:47.470 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:52:47.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:52:47.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:47.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:52:47.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:48.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:48.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:48.899 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:52:48.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:52:48.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:48.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:52:49.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:50.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:50.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:50.302 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:52:50.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:52:50.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:50.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:52:50.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:51.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:51.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:51.771 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:52:51.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:52:51.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:51.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:52:52.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:53.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:53.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:53.132 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:52:53.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:52:53.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:53.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:52:53.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:54.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:54.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:54.537 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:52:54.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:52:54.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:54.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:52:55.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:56.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:56.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:56.043 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:52:56.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:52:56.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:56.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:52:56.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:57.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:57.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:57.440 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:52:57.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:52:57.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:57.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:52:58.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:52:59.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:52:59.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:52:59.259 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:52:59.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:52:59.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:52:59.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:52:59.971 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:00.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:00.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:00.972 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:53:00.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:53:00.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:00.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:53:01.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:02.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:02.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:02.728 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:53:02.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:53:02.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:02.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:53:03.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:04.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:04.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:04.476 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:53:04.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:53:04.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:04.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:53:05.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:06.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:06.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:06.261 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:53:06.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:53:06.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:06.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:53:07.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:08.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:08.280 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:08.280 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:53:08.280 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:53:08.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:08.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:53:08.676 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/142870,v1:127.0.0.1:6803/142870] [v2:127.0.0.1:6804/142870,v1:127.0.0.1:6805/142870] exists,up 4cadedb9-d5a3-4dd3-b619-bdd5357aba48 2024-09-18T11:53:08.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:53:08.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:53:08.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:53:08.677 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: get_asok_path osd.0 2024-09-18T11:53:08.677 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:53:08.677 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:53:08.677 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:53:08.677 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:53:08.678 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:53:08.678 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-osd.0.asok 2024-09-18T11:53:08.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: CEPH_ARGS= 2024-09-18T11:53:08.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.76995/ceph-osd.0.asok config get osd_max_backfills 2024-09-18T11:53:08.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-09-18T11:53:08.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: echo '{"osd_max_backfills":"1"}' 2024-09-18T11:53:08.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: grep --quiet osd_max_backfills 2024-09-18T11:53:08.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:757: test_run_osd: run_osd td/ceph-helpers 1 --osd-max-backfills 20 2024-09-18T11:53:08.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:53:08.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:53:08.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:53:08.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:53:08.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-18T11:53:08.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:53:08.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:53:08.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:53:08.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:53:08.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-18T11:53:08.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-18T11:53:08.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:53:08.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:53:08.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:53:08.762 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:53:08.762 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:53:08.763 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:53:08.763 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:53:08.763 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:53:08.763 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:53:08.763 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:53:08.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:53:08.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:53:08.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:53:08.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:53:08.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:53:08.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:53:08.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:53:08.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:53:08.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:53:08.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:53:08.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:53:08.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-max-backfills 20' 2024-09-18T11:53:08.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-18T11:53:08.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:53:08.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=801d3e49-805a-439b-a585-91c1ff4502e6 2024-09-18T11:53:08.764 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 801d3e49-805a-439b-a585-91c1ff4502e6 2024-09-18T11:53:08.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 801d3e49-805a-439b-a585-91c1ff4502e6' 2024-09-18T11:53:08.765 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:53:08.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAkv+pmAkosLhAAyJzJbYhO+h3JpP4J3S9e2Q== 2024-09-18T11:53:08.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAkv+pmAkosLhAAyJzJbYhO+h3JpP4J3S9e2Q=="}' 2024-09-18T11:53:08.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 801d3e49-805a-439b-a585-91c1ff4502e6 -i td/ceph-helpers/1/new.json 2024-09-18T11:53:09.343 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:53:09.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-18T11:53:09.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 --mkfs --key AQAkv+pmAkosLhAAyJzJbYhO+h3JpP4J3S9e2Q== --osd-uuid 801d3e49-805a-439b-a585-91c1ff4502e6 2024-09-18T11:53:09.380 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:09.378+0000 7f8500287440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:53:09.398 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:09.397+0000 7f8500287440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:53:09.400 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:09.398+0000 7f8500287440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:53:09.400 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:09.398+0000 7f8500287440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:53:09.400 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:09.398+0000 7f8500287440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:53:09.400 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:09.398+0000 7f8500287440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:53:09.468 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:09.467+0000 7f8500287440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-18T11:53:15.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-18T11:53:15.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:53:15.648 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T11:53:15.648 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:53:15.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:53:16.137 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T11:53:16.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:53:16.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-09-18T11:53:16.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:53:16.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:53:16.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:53:16.160 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:16.158+0000 7f6cbf21f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:53:16.162 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:16.161+0000 7f6cbf21f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:53:16.163 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:16.162+0000 7f6cbf21f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:53:16.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:53:16.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:53:16.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:53:16.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:53:16.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:53:16.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:16.489 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:53:16.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:53:16.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:16.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:16.681 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:16.679+0000 7f6cbf21f440 -1 Falling back to public interface 2024-09-18T11:53:16.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:17.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:17.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:17.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:53:17.842 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:53:17.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:17.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:18.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:19.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:19.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:19.207 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:53:19.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:53:19.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:19.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:19.270 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:19.268+0000 7f6cbf21f440 -1 osd.1 0 log_to_monitors true 2024-09-18T11:53:19.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:20.603 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:20.603 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:20.603 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:53:20.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:53:20.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:20.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:21.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:22.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:22.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:22.069 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:53:22.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:53:22.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:22.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:22.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:23.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:23.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:23.431 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:53:23.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:53:23.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:23.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:23.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:24.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:24.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:24.805 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:53:24.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:53:24.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:24.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:25.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:26.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:26.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:26.278 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:53:26.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:53:26.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:26.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:26.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:27.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:27.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:27.710 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:53:27.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:53:27.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:27.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:28.280 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:29.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:29.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:29.282 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:53:29.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:53:29.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:29.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:29.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:30.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:30.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:30.695 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:53:30.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:53:30.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:30.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:31.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:32.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:32.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:32.220 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:53:32.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:53:32.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:32.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:33.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:34.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:34.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:34.248 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:53:34.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:53:34.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:34.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:34.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:35.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:35.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:35.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:53:35.748 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:53:35.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:35.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:36.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:37.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:37.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:37.911 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:53:37.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:53:37.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:37.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:39.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:40.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:40.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:40.050 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:53:40.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:53:40.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:40.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:40.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:41.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:41.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:41.504 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:53:41.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:53:41.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:41.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:42.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:43.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:43.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:43.227 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:53:43.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:53:43.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:43.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:53:43.598 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/144604,v1:127.0.0.1:6811/144604] [v2:127.0.0.1:6812/144604,v1:127.0.0.1:6813/144604] exists,up 801d3e49-805a-439b-a585-91c1ff4502e6 2024-09-18T11:53:43.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:53:43.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:53:43.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:53:43.599 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: get_asok_path osd.1 2024-09-18T11:53:43.599 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T11:53:43.599 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T11:53:43.600 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:53:43.600 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:53:43.600 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:53:43.600 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-osd.1.asok 2024-09-18T11:53:43.600 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: CEPH_ARGS= 2024-09-18T11:53:43.600 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.76995/ceph-osd.1.asok config get osd_max_backfills 2024-09-18T11:53:43.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-09-18T11:53:43.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:760: test_run_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-09-18T11:53:43.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: CEPH_ARGS='--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2024-09-18T11:53:43.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: run_osd td/ceph-helpers 2 2024-09-18T11:53:43.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:53:43.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:53:43.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:53:43.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:53:43.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-09-18T11:53:43.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2024-09-18T11:53:43.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:53:43.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:53:43.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:53:43.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-09-18T11:53:43.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-09-18T11:53:43.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:53:43.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:53:43.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:53:43.684 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:53:43.684 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:53:43.684 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:53:43.684 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:53:43.684 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:53:43.685 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:53:43.685 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:53:43.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:53:43.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:53:43.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:53:43.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:53:43.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:53:43.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:53:43.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:53:43.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:53:43.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:53:43.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:53:43.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:53:43.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:53:43.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-09-18T11:53:43.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:53:43.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cefae920-ae62-4d6b-a238-5a4dbaca61dd 2024-09-18T11:53:43.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cefae920-ae62-4d6b-a238-5a4dbaca61dd' 2024-09-18T11:53:43.686 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 cefae920-ae62-4d6b-a238-5a4dbaca61dd 2024-09-18T11:53:43.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:53:43.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBHv+pmuNOiKRAA+fE41KaApjnRXzntsGN3Gg== 2024-09-18T11:53:43.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBHv+pmuNOiKRAA+fE41KaApjnRXzntsGN3Gg=="}' 2024-09-18T11:53:43.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cefae920-ae62-4d6b-a238-5a4dbaca61dd -i td/ceph-helpers/2/new.json 2024-09-18T11:53:44.290 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:53:44.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-09-18T11:53:44.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHv+pmuNOiKRAA+fE41KaApjnRXzntsGN3Gg== --osd-uuid cefae920-ae62-4d6b-a238-5a4dbaca61dd 2024-09-18T11:53:44.329 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:44.327+0000 7fe73780e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:53:44.351 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:44.350+0000 7fe73780e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:53:44.352 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:44.351+0000 7fe73780e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:53:44.353 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:44.351+0000 7fe73780e440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-18T11:53:44.353 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:44.351+0000 7fe73780e440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-18T11:53:44.353 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:44.351+0000 7fe73780e440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-18T11:53:44.424 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:44.422+0000 7fe73780e440 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-09-18T11:53:50.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-09-18T11:53:50.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:53:50.337 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-18T11:53:50.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:53:50.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:53:50.829 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-18T11:53:50.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:53:50.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:53:50.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:53:50.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:53:50.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:53:50.853 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:50.852+0000 7f3dbf11d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:53:50.855 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:50.854+0000 7f3dbf11d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:53:50.856 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:50.855+0000 7f3dbf11d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:53:51.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:53:51.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:53:51.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:53:51.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:53:51.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:53:51.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:51.185 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:53:51.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:53:51.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:51.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:53:51.376 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:51.374+0000 7f3dbf11d440 -1 Falling back to public interface 2024-09-18T11:53:51.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:52.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:52.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:52.545 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:53:52.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:53:52.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:52.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:53:52.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:53.909 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:53:53.907+0000 7f3dbf11d440 -1 osd.2 0 log_to_monitors true 2024-09-18T11:53:53.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:53.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:53.911 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:53:53.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:53:53.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:53.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:53:54.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:55.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:55.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:55.283 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:53:55.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:53:55.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:55.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:53:55.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:56.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:56.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:56.644 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:53:56.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:53:56.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:56.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:53:57.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:58.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:58.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:58.231 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:53:58.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:53:58.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:58.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:53:58.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:53:59.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:53:59.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:53:59.729 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:53:59.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:53:59.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:53:59.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:54:00.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:01.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:01.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:01.158 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:54:01.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:54:01.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:01.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:54:01.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:02.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:02.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:02.539 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:54:02.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:54:02.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:02.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:54:02.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:03.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:03.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:03.944 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:54:03.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:54:03.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:03.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:54:04.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:05.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:05.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:05.333 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:54:05.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:54:05.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:05.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:54:05.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:06.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:06.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:06.874 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:54:06.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:54:06.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:06.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:54:07.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:08.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:08.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:08.689 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:54:08.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:54:08.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:08.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:54:09.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:10.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:10.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:10.741 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:54:10.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:54:10.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:10.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:54:11.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:12.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:12.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:12.229 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:54:12.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:54:12.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:12.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:54:12.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:13.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:13.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:13.968 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:54:13.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:54:13.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:13.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:54:14.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:15.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:15.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:15.492 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:54:15.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:54:15.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:15.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:54:16.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:17.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:17.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:17.197 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:54:17.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:54:17.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:17.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:54:17.675 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/146361,v1:127.0.0.1:6819/146361] [v2:127.0.0.1:6820/146361,v1:127.0.0.1:6821/146361] exists,up cefae920-ae62-4d6b-a238-5a4dbaca61dd 2024-09-18T11:54:17.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:54:17.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:54:17.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:54:17.677 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: get_asok_path osd.2 2024-09-18T11:54:17.677 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T11:54:17.677 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T11:54:17.677 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:54:17.677 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:17.677 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:17.677 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-osd.2.asok 2024-09-18T11:54:17.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: CEPH_ARGS= 2024-09-18T11:54:17.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.76995/ceph-osd.2.asok config get osd_max_backfills 2024-09-18T11:54:17.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: local 'backfills={"osd_max_backfills":"30"}' 2024-09-18T11:54:17.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:765: test_run_osd: test '{"osd_max_backfills":"30"}' = '{"osd_max_backfills":"30"}' 2024-09-18T11:54:17.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:767: test_run_osd: teardown td/ceph-helpers 2024-09-18T11:54:17.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:54:17.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:54:17.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:54:17.757 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:54:17.757 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:54:17.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:54:17.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:54:17.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:54:17.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:54:17.865 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:54:17.865 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:54:17.866 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:54:17.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:54:17.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:54:17.867 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:54:17.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:17.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:54:17.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:54:17.868 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:17.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:54:17.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:54:17.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:54:17.882 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:54:17.882 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:17.882 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:17.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:54:17.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:54:17.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:54:17.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:54:17.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_set_config td/ceph-helpers 2024-09-18T11:54:17.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: test_set_config: local dir=td/ceph-helpers 2024-09-18T11:54:17.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: test_set_config: setup td/ceph-helpers 2024-09-18T11:54:17.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:54:17.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:54:17.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:54:17.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:54:17.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:54:17.885 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:54:17.885 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:54:17.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:54:17.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:54:17.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:54:17.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:54:17.886 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:54:17.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:54:17.886 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:54:17.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:54:17.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:54:17.887 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:54:17.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:17.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:54:17.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:54:17.889 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:17.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:54:17.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:54:17.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:54:17.891 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:54:17.891 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:17.891 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:17.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:54:17.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:54:17.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:54:17.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:54:17.894 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:54:17.894 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:17.894 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:17.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:54:17.895 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:54:17.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:54:17.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:54:17.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:54:17.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: test_set_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:54:17.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:54:17.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:54:17.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:54:17.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:54:17.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:54:17.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:54:18.264 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:54:18.264 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:54:18.264 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:54:18.264 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:54:18.264 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:18.264 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:18.265 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:54:18.265 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:54:18.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:54:18.525 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:54:18.525 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:54:18.525 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:54:18.525 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:54:18.525 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:54:18.525 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:54:18.525 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:54:18.525 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:54:18.525 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:54:18.525 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:18.525 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:18.526 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:54:18.526 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:54:18.526 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:54:18.600 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:54:18.600 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:54:18.600 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:54:18.601 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:54:18.601 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:54:18.601 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:54:18.601 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:54:18.601 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:54:18.601 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:54:18.601 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:18.601 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:18.602 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:54:18.602 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:54:18.602 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:54:18.674 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: get_config mon a ms_crc_header 2024-09-18T11:54:18.674 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:54:18.674 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:54:18.674 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-09-18T11:54:18.675 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-09-18T11:54:18.675 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:54:18.675 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:54:18.675 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:54:18.676 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:54:18.676 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:18.676 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:18.676 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:54:18.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:54:18.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get ms_crc_header 2024-09-18T11:54:18.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: test true = true 2024-09-18T11:54:18.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1177: test_set_config: set_config mon a ms_crc_header false 2024-09-18T11:54:18.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2024-09-18T11:54:18.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2024-09-18T11:54:18.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2024-09-18T11:54:18.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=false 2024-09-18T11:54:18.750 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T11:54:18.750 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2024-09-18T11:54:18.750 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:54:18.751 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:54:18.751 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:54:18.751 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:18.751 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:18.751 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:54:18.751 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config set ms_crc_header false 2024-09-18T11:54:18.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T11:54:18.827 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: get_config mon a ms_crc_header 2024-09-18T11:54:18.827 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:54:18.827 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:54:18.827 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-09-18T11:54:18.827 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-09-18T11:54:18.827 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:54:18.827 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:54:18.827 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:54:18.827 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:54:18.827 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:18.827 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:18.828 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:54:18.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:54:18.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get ms_crc_header 2024-09-18T11:54:18.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: test false = false 2024-09-18T11:54:18.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1179: test_set_config: set_config mon a ms_crc_header true 2024-09-18T11:54:18.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2024-09-18T11:54:18.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2024-09-18T11:54:18.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2024-09-18T11:54:18.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T11:54:18.901 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T11:54:18.902 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2024-09-18T11:54:18.902 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:54:18.902 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:54:18.902 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:54:18.902 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:18.902 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:18.902 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:54:18.902 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config set ms_crc_header true 2024-09-18T11:54:18.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T11:54:18.976 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: get_config mon a ms_crc_header 2024-09-18T11:54:18.976 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:54:18.976 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:54:18.976 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-09-18T11:54:18.976 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-09-18T11:54:18.976 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:54:18.976 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:54:18.977 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:54:18.977 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:54:18.977 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:18.977 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:18.977 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:54:18.977 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:54:18.977 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get ms_crc_header 2024-09-18T11:54:19.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: test true = true 2024-09-18T11:54:19.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1181: test_set_config: teardown td/ceph-helpers 2024-09-18T11:54:19.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:54:19.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:54:19.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:54:19.052 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:54:19.052 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:54:19.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:54:19.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:54:19.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:54:19.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:54:19.154 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:54:19.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:54:19.155 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:54:19.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:54:19.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:54:19.157 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:54:19.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:19.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:54:19.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:54:19.159 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:19.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:54:19.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:54:19.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:54:19.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:54:19.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:19.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:19.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:54:19.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:54:19.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:54:19.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:54:19.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_setup td/ceph-helpers 2024-09-18T11:54:19.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:144: test_setup: local dir=td/ceph-helpers 2024-09-18T11:54:19.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:145: test_setup: setup td/ceph-helpers 2024-09-18T11:54:19.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:54:19.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:54:19.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:54:19.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:54:19.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:54:19.164 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:54:19.164 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:54:19.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:54:19.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:54:19.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:54:19.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:54:19.166 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:54:19.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:54:19.167 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:54:19.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:54:19.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:54:19.167 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:54:19.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:19.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:54:19.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:54:19.168 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:19.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:54:19.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:54:19.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:54:19.170 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:54:19.170 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:19.170 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:19.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:54:19.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:54:19.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:54:19.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:54:19.172 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:54:19.172 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:19.172 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:19.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:54:19.175 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:54:19.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:54:19.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:54:19.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:54:19.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:146: test_setup: test -d td/ceph-helpers 2024-09-18T11:54:19.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:147: test_setup: setup td/ceph-helpers 2024-09-18T11:54:19.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:54:19.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:54:19.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:54:19.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:54:19.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:54:19.176 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:54:19.176 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:54:19.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:54:19.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:54:19.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:54:19.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:54:19.177 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:54:19.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:54:19.177 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:54:19.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:54:19.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:54:19.178 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:54:19.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:19.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:54:19.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:54:19.180 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:19.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:54:19.181 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:54:19.181 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:54:19.182 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:54:19.182 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:19.182 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:19.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:54:19.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:54:19.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:54:19.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:54:19.184 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:54:19.184 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:19.184 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:19.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:54:19.187 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:54:19.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:54:19.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:54:19.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:54:19.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:148: test_setup: test -d td/ceph-helpers 2024-09-18T11:54:19.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:149: test_setup: teardown td/ceph-helpers 2024-09-18T11:54:19.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:54:19.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:54:19.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:54:19.188 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:54:19.188 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:54:19.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:54:19.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:54:19.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:54:19.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:54:19.189 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:54:19.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:54:19.189 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:54:19.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:54:19.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:54:19.191 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:54:19.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:19.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:54:19.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:54:19.192 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:19.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:54:19.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:54:19.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:54:19.195 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:54:19.195 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:19.195 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:19.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:54:19.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:54:19.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:54:19.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:54:19.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_teardown td/ceph-helpers 2024-09-18T11:54:19.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:220: test_teardown: local dir=td/ceph-helpers 2024-09-18T11:54:19.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221: test_teardown: setup td/ceph-helpers 2024-09-18T11:54:19.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:54:19.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:54:19.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:54:19.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:54:19.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:54:19.198 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:54:19.198 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:54:19.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:54:19.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:54:19.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:54:19.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:54:19.199 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:54:19.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:54:19.200 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:54:19.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:54:19.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:54:19.200 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:54:19.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:19.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:54:19.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:54:19.202 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:19.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:54:19.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:54:19.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:54:19.204 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:54:19.204 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:19.204 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:19.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:54:19.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:54:19.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:54:19.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:54:19.206 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:54:19.206 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:19.206 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:19.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:54:19.209 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:54:19.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:54:19.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:54:19.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:54:19.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:222: test_teardown: teardown td/ceph-helpers 2024-09-18T11:54:19.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:54:19.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:54:19.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:54:19.210 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:54:19.210 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:54:19.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:54:19.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:54:19.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:54:19.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:54:19.211 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:54:19.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:54:19.211 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:54:19.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:54:19.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:54:19.212 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:54:19.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:19.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:54:19.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:54:19.213 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:19.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:54:19.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:54:19.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:54:19.216 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:54:19.217 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:19.217 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:19.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:54:19.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:54:19.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:54:19.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:223: test_teardown: test -d td/ceph-helpers 2024-09-18T11:54:19.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:54:19.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_wait_background td/ceph-helpers 2024-09-18T11:54:19.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: test_wait_background: local pids= 2024-09-18T11:54:19.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 1' 2024-09-18T11:54:19.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T11:54:19.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T11:54:19.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 147527"' 2024-09-18T11:54:19.220 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 147527' 2024-09-18T11:54:19.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2024-09-18T11:54:19.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T11:54:19.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T11:54:19.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 147529"' 2024-09-18T11:54:19.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: bash -c 'sleep 1; exit 1' 2024-09-18T11:54:19.221 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 147529' 2024-09-18T11:54:19.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: test_wait_background: wait_background pids 2024-09-18T11:54:19.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 147527 147529' 2024-09-18T11:54:19.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T11:54:19.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T11:54:19.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 147527 2024-09-18T11:54:19.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/147530: /' 2024-09-18T11:54:19.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: bash -c 'sleep 2; exit 0' 2024-09-18T11:54:19.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/147532: /' 2024-09-18T11:54:20.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 1 2024-09-18T11:54:20.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: wait_background: return_code=1 2024-09-18T11:54:20.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T11:54:20.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 147529 2024-09-18T11:54:21.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T11:54:21.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T11:54:21.222 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T11:54:21.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 1 2024-09-18T11:54:21.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: test_wait_background: '[' 1 -ne 1 ']' 2024-09-18T11:54:21.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 0' 2024-09-18T11:54:21.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T11:54:21.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T11:54:21.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 147535"' 2024-09-18T11:54:21.223 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 147535' 2024-09-18T11:54:21.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2024-09-18T11:54:21.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T11:54:21.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T11:54:21.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: bash -c 'sleep 1; exit 0' 2024-09-18T11:54:21.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 147537"' 2024-09-18T11:54:21.223 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 147537' 2024-09-18T11:54:21.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: test_wait_background: wait_background pids 2024-09-18T11:54:21.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/147538: /' 2024-09-18T11:54:21.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 147535 147537' 2024-09-18T11:54:21.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T11:54:21.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T11:54:21.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 147535 2024-09-18T11:54:21.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: bash -c 'sleep 2; exit 0' 2024-09-18T11:54:21.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/147540: /' 2024-09-18T11:54:22.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T11:54:22.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T11:54:22.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 147537 2024-09-18T11:54:23.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T11:54:23.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T11:54:23.226 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T11:54:23.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T11:54:23.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2161: test_wait_background: '[' 0 -ne 0 ']' 2024-09-18T11:54:23.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: test_wait_background: '[' '!' -z '' ']' 2024-09-18T11:54:23.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:54:23.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_wait_for_clean td/ceph-helpers 2024-09-18T11:54:23.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1677: test_wait_for_clean: local dir=td/ceph-helpers 2024-09-18T11:54:23.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1679: test_wait_for_clean: setup td/ceph-helpers 2024-09-18T11:54:23.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:54:23.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:54:23.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:54:23.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:54:23.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:54:23.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:54:23.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:54:23.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:54:23.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:54:23.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:54:23.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:54:23.228 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:54:23.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:54:23.229 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:54:23.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:54:23.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:54:23.230 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:54:23.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:23.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:54:23.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:54:23.232 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:54:23.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:54:23.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:54:23.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:54:23.235 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:54:23.236 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:23.236 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:23.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:54:23.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:54:23.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:54:23.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:54:23.239 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:54:23.239 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:23.239 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:23.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:54:23.241 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:54:23.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:54:23.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:54:23.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:54:23.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1680: test_wait_for_clean: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-18T11:54:23.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:54:23.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:54:23.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:54:23.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:54:23.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:54:23.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-18T11:54:23.611 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:54:23.612 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:54:23.612 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:54:23.612 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:54:23.612 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:23.612 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:23.612 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:54:23.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-18T11:54:23.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:54:23.908 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:54:23.908 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:54:23.908 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:54:23.908 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:54:23.909 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:54:23.909 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:54:23.909 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:54:23.909 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:54:23.909 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:54:23.909 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:23.909 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:23.909 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:54:23.909 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:54:23.909 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:54:23.966 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:54:23.966 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:54:23.966 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:54:23.967 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:54:23.967 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:54:23.967 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:54:23.967 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:54:23.967 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:54:23.968 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:54:23.968 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:23.968 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:23.968 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:54:23.968 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:54:23.968 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:54:24.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1681: test_wait_for_clean: run_osd td/ceph-helpers 0 2024-09-18T11:54:24.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:54:24.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:54:24.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:54:24.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:54:24.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:54:24.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:54:24.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:54:24.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:54:24.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:54:24.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:54:24.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:54:24.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:54:24.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:54:24.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:54:24.047 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:54:24.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:54:24.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:54:24.048 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:54:24.048 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:24.048 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:24.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:54:24.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:54:24.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:54:24.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:54:24.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:54:24.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:54:24.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:54:24.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:54:24.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:54:24.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:54:24.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:54:24.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:54:24.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:54:24.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:54:24.049 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:54:24.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4eeb8ca1-fd1e-45ee-88da-7970576698ae 2024-09-18T11:54:24.050 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 4eeb8ca1-fd1e-45ee-88da-7970576698ae 2024-09-18T11:54:24.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4eeb8ca1-fd1e-45ee-88da-7970576698ae' 2024-09-18T11:54:24.050 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:54:24.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBwv+pmJK2mAxAAxPAaRWoVKP8o7pXXKyM2HA== 2024-09-18T11:54:24.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBwv+pmJK2mAxAAxPAaRWoVKP8o7pXXKyM2HA=="}' 2024-09-18T11:54:24.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4eeb8ca1-fd1e-45ee-88da-7970576698ae -i td/ceph-helpers/0/new.json 2024-09-18T11:54:24.930 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:54:24.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:54:24.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBwv+pmJK2mAxAAxPAaRWoVKP8o7pXXKyM2HA== --osd-uuid 4eeb8ca1-fd1e-45ee-88da-7970576698ae 2024-09-18T11:54:24.964 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:54:24.963+0000 7fc43ed1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:54:24.979 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:54:24.978+0000 7fc43ed1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:54:24.980 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:54:24.979+0000 7fc43ed1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:54:24.980 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:54:24.979+0000 7fc43ed1d440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:54:24.981 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:54:24.979+0000 7fc43ed1d440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:54:24.981 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:54:24.979+0000 7fc43ed1d440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:54:25.019 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:54:25.018+0000 7fc43ed1d440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:54:30.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:54:30.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:54:30.671 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:54:30.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:54:30.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:54:30.869 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:54:30.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:54:30.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:54:30.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:54:30.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:54:30.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:54:30.891 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:54:30.890+0000 7fe1f511f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:54:30.904 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:54:30.903+0000 7fe1f511f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:54:30.905 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:54:30.904+0000 7fe1f511f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:54:31.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:54:31.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:54:31.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:54:31.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:54:31.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:54:31.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:31.076 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:54:31.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:54:31.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:31.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:31.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:31.436 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:54:31.435+0000 7fe1f511f440 -1 Falling back to public interface 2024-09-18T11:54:32.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:32.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:32.262 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:54:32.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:54:32.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:32.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:32.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:33.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:33.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:33.445 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:54:33.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:54:33.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:33.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:33.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:33.988 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:54:33.987+0000 7fe1f511f440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:54:34.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:34.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:34.637 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:54:34.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:54:34.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:34.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:34.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:35.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:35.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:35.968 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:54:35.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:54:35.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:35.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:36.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:37.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:37.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:37.156 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:54:37.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:54:37.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:37.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:37.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:38.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:38.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:38.344 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:54:38.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:54:38.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:38.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:38.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:39.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:39.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:39.532 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:54:39.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:54:39.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:39.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:39.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:40.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:40.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:40.720 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:54:40.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:54:40.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:40.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:40.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:41.912 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:41.912 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:41.913 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:54:41.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:54:41.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:41.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:42.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:43.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:43.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:43.101 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:54:43.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:54:43.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:43.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:43.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:44.289 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:44.289 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:44.289 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:54:44.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:54:44.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:44.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:44.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:45.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:45.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:45.479 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:54:45.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:54:45.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:45.480 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:45.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:46.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:46.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:46.666 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:54:46.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:54:46.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:46.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:46.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:47.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:47.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:47.853 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:54:47.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:54:47.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:47.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:48.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:49.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:49.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:49.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:54:49.043 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:54:49.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:49.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:50.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:50.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:50.221 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:54:50.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:54:50.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:50.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:50.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:51.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:51.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:51.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:54:51.406 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:54:51.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:51.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:51.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:52.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:52.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:52.596 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T11:54:52.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:54:52.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:52.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:52.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:53.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:53.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:53.780 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-18T11:54:53.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T11:54:53.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:53.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:53.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:54.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:54.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:54.969 INFO:tasks.workunit.client.0.smithi157.stdout:20 2024-09-18T11:54:54.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T11:54:54.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:54.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:55.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:54:56.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:54:56.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:54:56.176 INFO:tasks.workunit.client.0.smithi157.stdout:21 2024-09-18T11:54:56.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-18T11:54:56.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:54:56.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:54:56.387 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/148478,v1:127.0.0.1:6801/148478] [v2:127.0.0.1:6802/148478,v1:127.0.0.1:6803/148478] exists,up 4eeb8ca1-fd1e-45ee-88da-7970576698ae 2024-09-18T11:54:56.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:54:56.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:54:56.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:54:56.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1682: test_wait_for_clean: run_mgr td/ceph-helpers x 2024-09-18T11:54:56.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:54:56.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:54:56.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:54:56.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:54:56.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:54:56.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:54:56.700 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:54:56.701 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:54:56.701 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:54:56.701 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:54:56.701 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:54:56.701 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:54:56.701 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:54:56.701 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:54:56.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:54:56.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: test_wait_for_clean: create_rbd_pool 2024-09-18T11:54:56.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:54:56.945 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:54:56.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:54:56.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:54:57.314 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:54:57.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:54:58.325 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:54:59.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: WAIT_FOR_CLEAN_TIMEOUT=1 2024-09-18T11:54:59.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: wait_for_clean 2024-09-18T11:54:59.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:54:59.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:54:59.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:54:59.403 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 1 .1 2024-09-18T11:54:59.404 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:54:59.404 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:54:59.404 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:54:59.404 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:54:59.404 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:54:59.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2024-09-18T11:54:59.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:54:59.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:54:59.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:54:59.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:54:59.429 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:54:59.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T11:54:59.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:54:59.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:54:59.612 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:54:59.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=0 2024-09-18T11:54:59.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 0 2024-09-18T11:54:59.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-0' 2024-09-18T11:54:59.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:54:59.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-0 2024-09-18T11:54:59.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:54:59.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:54:59.721 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-0 2024-09-18T11:54:59.721 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:54:59.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=0 2024-09-18T11:54:59.722 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 0 2024-09-18T11:54:59.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 0' 2024-09-18T11:54:59.723 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:54:59.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 0 -lt 0 2024-09-18T11:54:59.904 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:54:59.905 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:54:59.905 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:55:00.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-09-18T11:55:00.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-09-18T11:55:01.133 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:55:01.133 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:55:01.133 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:55:01.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:55:01.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:55:01.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:55:01.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:55:01.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:55:01.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:55:01.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:55:01.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:55:01.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T11:55:01.947 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:55:01.947 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:55:01.947 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:55:02.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-18T11:55:02.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T11:55:02.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T11:55:02.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T11:55:02.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:55:02.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T11:55:02.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:55:02.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:55:02.545 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:55:02.545 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:55:02.545 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:55:02.545 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:55:02.546 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:55:02.546 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:55:02.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T11:55:02.885 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:55:02.885 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:55:02.885 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:55:03.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-18T11:55:03.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T11:55:03.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T11:55:03.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T11:55:03.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T11:55:03.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T11:55:03.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T11:55:03.358 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T11:55:03.358 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T11:55:03.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T11:55:03.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T11:55:03.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 4 )) 2024-09-18T11:55:03.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:55:03.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T11:55:04.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:55:04.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:55:04.025 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:55:04.025 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:55:04.026 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:55:04.026 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:55:04.026 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:55:04.026 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:55:04.365 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T11:55:04.366 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:55:04.366 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:55:04.366 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:55:04.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-18T11:55:04.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T11:55:04.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T11:55:04.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T11:55:04.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T11:55:04.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T11:55:04.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T11:55:04.835 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T11:55:04.835 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T11:55:05.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T11:55:05.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T11:55:05.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 4 )) 2024-09-18T11:55:05.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:55:05.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-18T11:55:05.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:55:05.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:55:05.712 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:55:05.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:55:05.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:55:05.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:55:05.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:55:05.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:55:06.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T11:55:06.053 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:55:06.054 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:55:06.054 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:55:06.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-18T11:55:06.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T11:55:06.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T11:55:06.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T11:55:06.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T11:55:06.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T11:55:06.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T11:55:06.523 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T11:55:06.523 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T11:55:07.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T11:55:07.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T11:55:07.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 4 )) 2024-09-18T11:55:07.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:55:07.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.3 2024-09-18T11:55:07.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:55:07.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:55:07.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:55:07.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:55:07.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:55:07.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:55:07.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:55:07.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:55:07.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T11:55:07.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:55:07.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:55:07.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:55:08.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-18T11:55:08.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T11:55:08.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T11:55:08.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T11:55:08.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T11:55:08.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T11:55:08.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T11:55:08.154 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T11:55:08.154 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T11:55:08.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T11:55:08.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T11:55:08.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 4 )) 2024-09-18T11:55:08.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-18T11:55:09.105 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:55:09.105 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_fingerprint": "44bdbfc7-1108-4685-8723-f098208dfbe7", 2024-09-18T11:55:09.105 INFO:tasks.workunit.client.0.smithi157.stdout: "version": "17.2.7-1645-g7061ea6c", 2024-09-18T11:55:09.105 INFO:tasks.workunit.client.0.smithi157.stdout: "commit": "7061ea6ca50908d104e9f3979dc615e118d9e153", 2024-09-18T11:55:09.106 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-18T11:55:09.103585+0000", 2024-09-18T11:55:09.106 INFO:tasks.workunit.client.0.smithi157.stdout: "tag": "", 2024-09-18T11:55:09.106 INFO:tasks.workunit.client.0.smithi157.stdout: "health": { 2024-09-18T11:55:09.106 INFO:tasks.workunit.client.0.smithi157.stdout: "status": "HEALTH_WARN", 2024-09-18T11:55:09.106 INFO:tasks.workunit.client.0.smithi157.stdout: "checks": { 2024-09-18T11:55:09.106 INFO:tasks.workunit.client.0.smithi157.stdout: "PG_DEGRADED": { 2024-09-18T11:55:09.106 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-18T11:55:09.106 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-18T11:55:09.106 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded", 2024-09-18T11:55:09.106 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 2 2024-09-18T11:55:09.106 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.106 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-18T11:55:09.106 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.107 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2024-09-18T11:55:09.107 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.107 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.107 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0]" 2024-09-18T11:55:09.107 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.107 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.107 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-18T11:55:09.107 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.107 INFO:tasks.workunit.client.0.smithi157.stdout: "TOO_FEW_OSDS": { 2024-09-18T11:55:09.107 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-18T11:55:09.107 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-18T11:55:09.107 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2024-09-18T11:55:09.107 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 2 2024-09-18T11:55:09.107 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.108 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [], 2024-09-18T11:55:09.108 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-18T11:55:09.108 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.108 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.108 INFO:tasks.workunit.client.0.smithi157.stdout: "mutes": [] 2024-09-18T11:55:09.108 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.108 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_first_committed": 1, 2024-09-18T11:55:09.108 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_last_committed": 1, 2024-09-18T11:55:09.108 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap": { 2024-09-18T11:55:09.108 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-18T11:55:09.108 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "92255036-ee37-4797-aa7f-f8e6b3f0325b", 2024-09-18T11:55:09.108 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-18T11:54:23.260840Z", 2024-09-18T11:55:09.108 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-18T11:54:23.260840Z", 2024-09-18T11:55:09.109 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release": 17, 2024-09-18T11:55:09.109 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:55:09.109 INFO:tasks.workunit.client.0.smithi157.stdout: "election_strategy": 1, 2024-09-18T11:55:09.109 INFO:tasks.workunit.client.0.smithi157.stdout: "disallowed_leaders: ": "", 2024-09-18T11:55:09.109 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": false, 2024-09-18T11:55:09.109 INFO:tasks.workunit.client.0.smithi157.stdout: "tiebreaker_mon": "", 2024-09-18T11:55:09.109 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_ranks: ": "", 2024-09-18T11:55:09.109 INFO:tasks.workunit.client.0.smithi157.stdout: "features": { 2024-09-18T11:55:09.109 INFO:tasks.workunit.client.0.smithi157.stdout: "persistent": [ 2024-09-18T11:55:09.109 INFO:tasks.workunit.client.0.smithi157.stdout: "kraken", 2024-09-18T11:55:09.109 INFO:tasks.workunit.client.0.smithi157.stdout: "luminous", 2024-09-18T11:55:09.109 INFO:tasks.workunit.client.0.smithi157.stdout: "mimic", 2024-09-18T11:55:09.109 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap-prune", 2024-09-18T11:55:09.109 INFO:tasks.workunit.client.0.smithi157.stdout: "nautilus", 2024-09-18T11:55:09.109 INFO:tasks.workunit.client.0.smithi157.stdout: "octopus", 2024-09-18T11:55:09.110 INFO:tasks.workunit.client.0.smithi157.stdout: "pacific", 2024-09-18T11:55:09.110 INFO:tasks.workunit.client.0.smithi157.stdout: "elector-pinging", 2024-09-18T11:55:09.110 INFO:tasks.workunit.client.0.smithi157.stdout: "quincy" 2024-09-18T11:55:09.110 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.110 INFO:tasks.workunit.client.0.smithi157.stdout: "optional": [] 2024-09-18T11:55:09.110 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.110 INFO:tasks.workunit.client.0.smithi157.stdout: "mons": [ 2024-09-18T11:55:09.110 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.110 INFO:tasks.workunit.client.0.smithi157.stdout: "rank": 0, 2024-09-18T11:55:09.110 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "a", 2024-09-18T11:55:09.110 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-18T11:55:09.110 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-18T11:55:09.110 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.110 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-18T11:55:09.110 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7109", 2024-09-18T11:55:09.111 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 0 2024-09-18T11:55:09.111 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.111 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:55:09.111 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.111 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7109/0", 2024-09-18T11:55:09.111 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:7109/0", 2024-09-18T11:55:09.111 INFO:tasks.workunit.client.0.smithi157.stdout: "priority": 0, 2024-09-18T11:55:09.111 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 0, 2024-09-18T11:55:09.111 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_location": "{}" 2024-09-18T11:55:09.111 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.111 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:55:09.111 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.112 INFO:tasks.workunit.client.0.smithi157.stdout: "quorum": [ 2024-09-18T11:55:09.112 INFO:tasks.workunit.client.0.smithi157.stdout: 0 2024-09-18T11:55:09.112 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.112 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap": { 2024-09-18T11:55:09.112 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 13, 2024-09-18T11:55:09.112 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "92255036-ee37-4797-aa7f-f8e6b3f0325b", 2024-09-18T11:55:09.112 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-18T11:54:24.133139+0000", 2024-09-18T11:55:09.112 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-18T11:54:59.123204+0000", 2024-09-18T11:55:09.112 INFO:tasks.workunit.client.0.smithi157.stdout: "last_up_change": "2024-09-18T11:54:55.550719+0000", 2024-09-18T11:55:09.112 INFO:tasks.workunit.client.0.smithi157.stdout: "last_in_change": "2024-09-18T11:54:24.808820+0000", 2024-09-18T11:55:09.112 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T11:55:09.112 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_num": 5799936, 2024-09-18T11:55:09.112 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_set": [ 2024-09-18T11:55:09.113 INFO:tasks.workunit.client.0.smithi157.stdout: "pglog_hardlimit", 2024-09-18T11:55:09.113 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snapdirs", 2024-09-18T11:55:09.113 INFO:tasks.workunit.client.0.smithi157.stdout: "recovery_deletes", 2024-09-18T11:55:09.113 INFO:tasks.workunit.client.0.smithi157.stdout: "sortbitwise" 2024-09-18T11:55:09.113 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.113 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_version": 3, 2024-09-18T11:55:09.113 INFO:tasks.workunit.client.0.smithi157.stdout: "full_ratio": 0.99000000953674316, 2024-09-18T11:55:09.113 INFO:tasks.workunit.client.0.smithi157.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-18T11:55:09.113 INFO:tasks.workunit.client.0.smithi157.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T11:55:09.113 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_snapshot": "", 2024-09-18T11:55:09.113 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_max": 1, 2024-09-18T11:55:09.113 INFO:tasks.workunit.client.0.smithi157.stdout: "max_osd": 1, 2024-09-18T11:55:09.113 INFO:tasks.workunit.client.0.smithi157.stdout: "require_min_compat_client": "luminous", 2024-09-18T11:55:09.114 INFO:tasks.workunit.client.0.smithi157.stdout: "min_compat_client": "jewel", 2024-09-18T11:55:09.114 INFO:tasks.workunit.client.0.smithi157.stdout: "require_osd_release": "quincy", 2024-09-18T11:55:09.114 INFO:tasks.workunit.client.0.smithi157.stdout: "pools": [ 2024-09-18T11:55:09.114 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.114 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-18T11:55:09.114 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_name": "rbd", 2024-09-18T11:55:09.114 INFO:tasks.workunit.client.0.smithi157.stdout: "create_time": "2024-09-18T11:54:57.122914+0000", 2024-09-18T11:55:09.114 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": 8193, 2024-09-18T11:55:09.114 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T11:55:09.114 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 1, 2024-09-18T11:55:09.114 INFO:tasks.workunit.client.0.smithi157.stdout: "size": 2, 2024-09-18T11:55:09.114 INFO:tasks.workunit.client.0.smithi157.stdout: "min_size": 1, 2024-09-18T11:55:09.114 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_rule": 0, 2024-09-18T11:55:09.115 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_count": 0, 2024-09-18T11:55:09.115 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_target": 0, 2024-09-18T11:55:09.115 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T11:55:09.115 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T11:55:09.115 INFO:tasks.workunit.client.0.smithi157.stdout: "object_hash": 2, 2024-09-18T11:55:09.115 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_autoscale_mode": "off", 2024-09-18T11:55:09.115 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num": 4, 2024-09-18T11:55:09.115 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num": 4, 2024-09-18T11:55:09.115 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num_target": 4, 2024-09-18T11:55:09.115 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_target": 4, 2024-09-18T11:55:09.115 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_pending": 4, 2024-09-18T11:55:09.115 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pg_merge_meta": { 2024-09-18T11:55:09.115 INFO:tasks.workunit.client.0.smithi157.stdout: "source_pgid": "0.0", 2024-09-18T11:55:09.115 INFO:tasks.workunit.client.0.smithi157.stdout: "ready_epoch": 0, 2024-09-18T11:55:09.116 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_started": 0, 2024-09-18T11:55:09.116 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": 0, 2024-09-18T11:55:09.116 INFO:tasks.workunit.client.0.smithi157.stdout: "source_version": "0'0", 2024-09-18T11:55:09.116 INFO:tasks.workunit.client.0.smithi157.stdout: "target_version": "0'0" 2024-09-18T11:55:09.116 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.116 INFO:tasks.workunit.client.0.smithi157.stdout: "last_change": "13", 2024-09-18T11:55:09.116 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend": "0", 2024-09-18T11:55:09.116 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T11:55:09.116 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T11:55:09.116 INFO:tasks.workunit.client.0.smithi157.stdout: "auid": 0, 2024-09-18T11:55:09.116 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_mode": "selfmanaged", 2024-09-18T11:55:09.116 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_seq": 3, 2024-09-18T11:55:09.116 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_epoch": 13, 2024-09-18T11:55:09.117 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_snaps": [], 2024-09-18T11:55:09.117 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps": "[]", 2024-09-18T11:55:09.117 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_bytes": 0, 2024-09-18T11:55:09.117 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_objects": 0, 2024-09-18T11:55:09.117 INFO:tasks.workunit.client.0.smithi157.stdout: "tiers": [], 2024-09-18T11:55:09.117 INFO:tasks.workunit.client.0.smithi157.stdout: "tier_of": -1, 2024-09-18T11:55:09.117 INFO:tasks.workunit.client.0.smithi157.stdout: "read_tier": -1, 2024-09-18T11:55:09.117 INFO:tasks.workunit.client.0.smithi157.stdout: "write_tier": -1, 2024-09-18T11:55:09.117 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_mode": "none", 2024-09-18T11:55:09.117 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_bytes": 0, 2024-09-18T11:55:09.117 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_objects": 0, 2024-09-18T11:55:09.117 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T11:55:09.117 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T11:55:09.118 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T11:55:09.118 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_flush_age": 0, 2024-09-18T11:55:09.118 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_evict_age": 0, 2024-09-18T11:55:09.118 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profile": "", 2024-09-18T11:55:09.118 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_params": { 2024-09-18T11:55:09.118 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "none" 2024-09-18T11:55:09.118 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.118 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_period": 0, 2024-09-18T11:55:09.118 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_count": 0, 2024-09-18T11:55:09.118 INFO:tasks.workunit.client.0.smithi157.stdout: "use_gmt_hitset": true, 2024-09-18T11:55:09.118 INFO:tasks.workunit.client.0.smithi157.stdout: "min_read_recency_for_promote": 0, 2024-09-18T11:55:09.118 INFO:tasks.workunit.client.0.smithi157.stdout: "min_write_recency_for_promote": 0, 2024-09-18T11:55:09.118 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T11:55:09.119 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_search_last_n": 0, 2024-09-18T11:55:09.119 INFO:tasks.workunit.client.0.smithi157.stdout: "grade_table": [], 2024-09-18T11:55:09.119 INFO:tasks.workunit.client.0.smithi157.stdout: "stripe_width": 0, 2024-09-18T11:55:09.119 INFO:tasks.workunit.client.0.smithi157.stdout: "expected_num_objects": 0, 2024-09-18T11:55:09.119 INFO:tasks.workunit.client.0.smithi157.stdout: "fast_read": false, 2024-09-18T11:55:09.119 INFO:tasks.workunit.client.0.smithi157.stdout: "options": {}, 2024-09-18T11:55:09.119 INFO:tasks.workunit.client.0.smithi157.stdout: "application_metadata": { 2024-09-18T11:55:09.119 INFO:tasks.workunit.client.0.smithi157.stdout: "rbd": {} 2024-09-18T11:55:09.119 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.119 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.119 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.119 INFO:tasks.workunit.client.0.smithi157.stdout: "osds": [ 2024-09-18T11:55:09.119 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.119 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-18T11:55:09.120 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "4eeb8ca1-fd1e-45ee-88da-7970576698ae", 2024-09-18T11:55:09.120 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 1, 2024-09-18T11:55:09.120 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-18T11:55:09.120 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-18T11:55:09.120 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-18T11:55:09.120 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 0, 2024-09-18T11:55:09.120 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 0, 2024-09-18T11:55:09.120 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 5, 2024-09-18T11:55:09.120 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 7, 2024-09-18T11:55:09.120 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 0, 2024-09-18T11:55:09.120 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-18T11:55:09.120 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-18T11:55:09.120 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-18T11:55:09.121 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.121 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-18T11:55:09.121 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6800", 2024-09-18T11:55:09.121 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 148478 2024-09-18T11:55:09.121 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.121 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.121 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-18T11:55:09.121 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6801", 2024-09-18T11:55:09.121 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 148478 2024-09-18T11:55:09.121 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.121 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:55:09.121 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.121 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-18T11:55:09.121 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-18T11:55:09.122 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.122 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-18T11:55:09.122 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6802", 2024-09-18T11:55:09.122 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 148478 2024-09-18T11:55:09.122 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.122 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.122 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-18T11:55:09.122 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6803", 2024-09-18T11:55:09.122 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 148478 2024-09-18T11:55:09.122 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.122 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:55:09.122 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.122 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-18T11:55:09.122 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-18T11:55:09.122 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.123 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-18T11:55:09.123 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6806", 2024-09-18T11:55:09.123 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 148478 2024-09-18T11:55:09.123 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.123 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.123 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-18T11:55:09.123 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6807", 2024-09-18T11:55:09.123 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 148478 2024-09-18T11:55:09.123 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.123 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:55:09.123 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.123 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-18T11:55:09.123 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-18T11:55:09.123 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.124 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-18T11:55:09.124 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6804", 2024-09-18T11:55:09.124 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 148478 2024-09-18T11:55:09.124 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.124 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.124 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-18T11:55:09.124 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6805", 2024-09-18T11:55:09.124 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 148478 2024-09-18T11:55:09.124 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.124 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:55:09.124 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.124 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6801/148478", 2024-09-18T11:55:09.125 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6803/148478", 2024-09-18T11:55:09.125 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6807/148478", 2024-09-18T11:55:09.125 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6805/148478", 2024-09-18T11:55:09.125 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-18T11:55:09.125 INFO:tasks.workunit.client.0.smithi157.stdout: "exists", 2024-09-18T11:55:09.125 INFO:tasks.workunit.client.0.smithi157.stdout: "up" 2024-09-18T11:55:09.125 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:55:09.125 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.125 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.125 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_xinfo": [ 2024-09-18T11:55:09.125 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.125 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-18T11:55:09.125 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "0.000000", 2024-09-18T11:55:09.125 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-18T11:55:09.126 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-18T11:55:09.126 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540138320759226367, 2024-09-18T11:55:09.126 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-18T11:55:09.126 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-18T11:54:55.131564+0000", 2024-09-18T11:55:09.126 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-18T11:55:09.126 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.126 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.126 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap": [], 2024-09-18T11:55:09.126 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap_items": [], 2024-09-18T11:55:09.126 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_temp": [], 2024-09-18T11:55:09.126 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_temp": [], 2024-09-18T11:55:09.126 INFO:tasks.workunit.client.0.smithi157.stdout: "blocklist": {}, 2024-09-18T11:55:09.126 INFO:tasks.workunit.client.0.smithi157.stdout: "range_blocklist": {}, 2024-09-18T11:55:09.126 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profiles": { 2024-09-18T11:55:09.127 INFO:tasks.workunit.client.0.smithi157.stdout: "default": { 2024-09-18T11:55:09.127 INFO:tasks.workunit.client.0.smithi157.stdout: "k": "2", 2024-09-18T11:55:09.127 INFO:tasks.workunit.client.0.smithi157.stdout: "m": "2", 2024-09-18T11:55:09.127 INFO:tasks.workunit.client.0.smithi157.stdout: "plugin": "jerasure", 2024-09-18T11:55:09.127 INFO:tasks.workunit.client.0.smithi157.stdout: "technique": "reed_sol_van" 2024-09-18T11:55:09.127 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.127 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.127 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps_queue": [ 2024-09-18T11:55:09.127 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.127 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-18T11:55:09.127 INFO:tasks.workunit.client.0.smithi157.stdout: "snaps": [ 2024-09-18T11:55:09.127 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.127 INFO:tasks.workunit.client.0.smithi157.stdout: "begin": 2, 2024-09-18T11:55:09.127 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 1 2024-09-18T11:55:09.128 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.128 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:55:09.128 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.128 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.128 INFO:tasks.workunit.client.0.smithi157.stdout: "new_removed_snaps": [ 2024-09-18T11:55:09.128 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.128 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-18T11:55:09.128 INFO:tasks.workunit.client.0.smithi157.stdout: "snaps": [ 2024-09-18T11:55:09.128 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.128 INFO:tasks.workunit.client.0.smithi157.stdout: "begin": 2, 2024-09-18T11:55:09.128 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 1 2024-09-18T11:55:09.128 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.128 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:55:09.128 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.128 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.129 INFO:tasks.workunit.client.0.smithi157.stdout: "new_purged_snaps": [], 2024-09-18T11:55:09.129 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_node_flags": {}, 2024-09-18T11:55:09.129 INFO:tasks.workunit.client.0.smithi157.stdout: "device_class_flags": {}, 2024-09-18T11:55:09.129 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": { 2024-09-18T11:55:09.129 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_enabled": false, 2024-09-18T11:55:09.129 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_bucket_count": 0, 2024-09-18T11:55:09.129 INFO:tasks.workunit.client.0.smithi157.stdout: "degraded_stretch_mode": 0, 2024-09-18T11:55:09.129 INFO:tasks.workunit.client.0.smithi157.stdout: "recovering_stretch_mode": 0, 2024-09-18T11:55:09.129 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_bucket": 0 2024-09-18T11:55:09.129 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.129 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.129 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_metadata": [ 2024-09-18T11:55:09.129 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.129 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-18T11:55:09.130 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-18T11:55:09.130 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6802/148478,v1:127.0.0.1:6803/148478]", 2024-09-18T11:55:09.130 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-18T11:55:09.130 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-18T11:55:09.130 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-18T11:55:09.130 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T11:55:09.130 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-18T11:55:09.130 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T11:55:09.130 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T11:55:09.130 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T11:55:09.130 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T11:55:09.130 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T11:55:09.130 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T11:55:09.130 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2024-09-18T11:55:09.131 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T11:55:09.131 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T11:55:09.131 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T11:55:09.131 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T11:55:09.131 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T11:55:09.131 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "quincy", 2024-09-18T11:55:09.131 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T11:55:09.131 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T11:55:09.131 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T11:55:09.131 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-18T11:55:09.131 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-18T11:54:30.001913Z", 2024-09-18T11:55:09.131 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-18T11:55:09.131 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-18T11:55:09.131 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T11:55:09.131 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-18T11:55:09.132 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-18T11:55:09.132 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T11:55:09.132 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-18T11:55:09.132 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6800/148478,v1:127.0.0.1:6801/148478]", 2024-09-18T11:55:09.132 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-18T11:55:09.132 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/148478,v1:127.0.0.1:6807/148478]", 2024-09-18T11:55:09.132 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/148478,v1:127.0.0.1:6805/148478]", 2024-09-18T11:55:09.132 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-18T11:55:09.132 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-18T11:55:09.132 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T11:55:09.132 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T11:55:09.132 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-18T11:55:09.132 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487300", 2024-09-18T11:55:09.133 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T11:55:09.133 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T11:55:09.133 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-18T11:55:09.133 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/ceph-helpers/0", 2024-09-18T11:55:09.133 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-18T11:55:09.133 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-18T11:55:09.133 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-18T11:55:09.133 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.133 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.133 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_clean_epochs": { 2024-09-18T11:55:09.133 INFO:tasks.workunit.client.0.smithi157.stdout: "min_last_epoch_clean": 0, 2024-09-18T11:55:09.133 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": { 2024-09-18T11:55:09.133 INFO:tasks.workunit.client.0.smithi157.stdout: "per_pool": [] 2024-09-18T11:55:09.134 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.134 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_epochs": [ 2024-09-18T11:55:09.134 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.134 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-18T11:55:09.134 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 5 2024-09-18T11:55:09.134 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.134 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:55:09.134 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.134 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_first_committed": 1, 2024-09-18T11:55:09.134 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_last_committed": 13, 2024-09-18T11:55:09.134 INFO:tasks.workunit.client.0.smithi157.stdout: "crushmap": { 2024-09-18T11:55:09.134 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": [ 2024-09-18T11:55:09.134 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.134 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-18T11:55:09.135 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.0", 2024-09-18T11:55:09.135 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-18T11:55:09.135 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.135 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.135 INFO:tasks.workunit.client.0.smithi157.stdout: "types": [ 2024-09-18T11:55:09.135 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.135 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 0, 2024-09-18T11:55:09.135 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd" 2024-09-18T11:55:09.135 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.135 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.135 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-18T11:55:09.135 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "host" 2024-09-18T11:55:09.135 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.135 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.135 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 2, 2024-09-18T11:55:09.136 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "chassis" 2024-09-18T11:55:09.136 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.136 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.136 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 3, 2024-09-18T11:55:09.136 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "rack" 2024-09-18T11:55:09.136 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.136 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.136 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 4, 2024-09-18T11:55:09.136 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "row" 2024-09-18T11:55:09.136 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.136 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.136 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 5, 2024-09-18T11:55:09.136 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pdu" 2024-09-18T11:55:09.136 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.136 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.137 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 6, 2024-09-18T11:55:09.137 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pod" 2024-09-18T11:55:09.137 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.137 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.137 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 7, 2024-09-18T11:55:09.137 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "room" 2024-09-18T11:55:09.137 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.137 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.137 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 8, 2024-09-18T11:55:09.137 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "datacenter" 2024-09-18T11:55:09.137 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.137 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.137 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 9, 2024-09-18T11:55:09.137 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "zone" 2024-09-18T11:55:09.138 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.138 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.138 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 10, 2024-09-18T11:55:09.138 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "region" 2024-09-18T11:55:09.138 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.138 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.138 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-18T11:55:09.138 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "root" 2024-09-18T11:55:09.138 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.138 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.138 INFO:tasks.workunit.client.0.smithi157.stdout: "buckets": [ 2024-09-18T11:55:09.138 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.138 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -1, 2024-09-18T11:55:09.138 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default", 2024-09-18T11:55:09.139 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-18T11:55:09.139 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-18T11:55:09.139 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-18T11:55:09.139 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-18T11:55:09.139 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-18T11:55:09.139 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-18T11:55:09.139 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.139 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-18T11:55:09.139 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-18T11:55:09.139 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-18T11:55:09.139 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.139 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:55:09.139 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.139 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.139 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -2, 2024-09-18T11:55:09.140 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default~hdd", 2024-09-18T11:55:09.140 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-18T11:55:09.140 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-18T11:55:09.140 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-18T11:55:09.140 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-18T11:55:09.140 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-18T11:55:09.140 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-18T11:55:09.140 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.140 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-18T11:55:09.140 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-18T11:55:09.140 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-18T11:55:09.140 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.140 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:55:09.140 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.141 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.141 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-18T11:55:09.141 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157", 2024-09-18T11:55:09.141 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-18T11:55:09.141 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-18T11:55:09.141 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-18T11:55:09.141 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-18T11:55:09.141 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-18T11:55:09.141 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-18T11:55:09.141 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.141 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-18T11:55:09.141 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-18T11:55:09.141 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-18T11:55:09.141 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.142 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:55:09.142 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.142 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.142 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-18T11:55:09.142 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157~hdd", 2024-09-18T11:55:09.142 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-18T11:55:09.142 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-18T11:55:09.142 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-18T11:55:09.142 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-18T11:55:09.142 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-18T11:55:09.142 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-18T11:55:09.142 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.142 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-18T11:55:09.142 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-18T11:55:09.143 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-18T11:55:09.143 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.143 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:55:09.143 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.143 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.143 INFO:tasks.workunit.client.0.smithi157.stdout: "rules": [ 2024-09-18T11:55:09.143 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.143 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_id": 0, 2024-09-18T11:55:09.143 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_name": "replicated_rule", 2024-09-18T11:55:09.143 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 1, 2024-09-18T11:55:09.143 INFO:tasks.workunit.client.0.smithi157.stdout: "steps": [ 2024-09-18T11:55:09.143 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.143 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "take", 2024-09-18T11:55:09.143 INFO:tasks.workunit.client.0.smithi157.stdout: "item": -1, 2024-09-18T11:55:09.144 INFO:tasks.workunit.client.0.smithi157.stdout: "item_name": "default" 2024-09-18T11:55:09.144 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.144 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.144 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "choose_firstn", 2024-09-18T11:55:09.144 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 0, 2024-09-18T11:55:09.144 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "osd" 2024-09-18T11:55:09.144 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.144 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.144 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "emit" 2024-09-18T11:55:09.144 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.144 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:55:09.144 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.144 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.144 INFO:tasks.workunit.client.0.smithi157.stdout: "tunables": { 2024-09-18T11:55:09.145 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_tries": 0, 2024-09-18T11:55:09.145 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_fallback_tries": 0, 2024-09-18T11:55:09.145 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_total_tries": 50, 2024-09-18T11:55:09.145 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_descend_once": 1, 2024-09-18T11:55:09.145 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_vary_r": 1, 2024-09-18T11:55:09.145 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_stable": 1, 2024-09-18T11:55:09.145 INFO:tasks.workunit.client.0.smithi157.stdout: "straw_calc_version": 1, 2024-09-18T11:55:09.145 INFO:tasks.workunit.client.0.smithi157.stdout: "allowed_bucket_algs": 54, 2024-09-18T11:55:09.145 INFO:tasks.workunit.client.0.smithi157.stdout: "profile": "jewel", 2024-09-18T11:55:09.145 INFO:tasks.workunit.client.0.smithi157.stdout: "optimal_tunables": 1, 2024-09-18T11:55:09.145 INFO:tasks.workunit.client.0.smithi157.stdout: "legacy_tunables": 0, 2024-09-18T11:55:09.145 INFO:tasks.workunit.client.0.smithi157.stdout: "minimum_required_version": "jewel", 2024-09-18T11:55:09.145 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables": 1, 2024-09-18T11:55:09.145 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables2": 1, 2024-09-18T11:55:09.146 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v2_rules": 0, 2024-09-18T11:55:09.146 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables3": 1, 2024-09-18T11:55:09.146 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v3_rules": 0, 2024-09-18T11:55:09.146 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v4_buckets": 1, 2024-09-18T11:55:09.146 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables5": 1, 2024-09-18T11:55:09.146 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v5_rules": 0 2024-09-18T11:55:09.146 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.146 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_args": {} 2024-09-18T11:55:09.146 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.146 INFO:tasks.workunit.client.0.smithi157.stdout: "fsmap": { 2024-09-18T11:55:09.146 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-18T11:55:09.146 INFO:tasks.workunit.client.0.smithi157.stdout: "default_fscid": -1, 2024-09-18T11:55:09.146 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": { 2024-09-18T11:55:09.147 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": {}, 2024-09-18T11:55:09.147 INFO:tasks.workunit.client.0.smithi157.stdout: "ro_compat": {}, 2024-09-18T11:55:09.147 INFO:tasks.workunit.client.0.smithi157.stdout: "incompat": { 2024-09-18T11:55:09.147 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_1": "base v0.20", 2024-09-18T11:55:09.147 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_2": "client writeable ranges", 2024-09-18T11:55:09.147 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_3": "default file layouts on dirs", 2024-09-18T11:55:09.147 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_4": "dir inode in separate object", 2024-09-18T11:55:09.147 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_5": "mds uses versioned encoding", 2024-09-18T11:55:09.147 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-18T11:55:09.147 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_8": "no anchor table", 2024-09-18T11:55:09.147 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_9": "file layout v2", 2024-09-18T11:55:09.147 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_10": "snaprealm v2" 2024-09-18T11:55:09.147 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.147 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.147 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_flags": { 2024-09-18T11:55:09.148 INFO:tasks.workunit.client.0.smithi157.stdout: "enable_multiple": true, 2024-09-18T11:55:09.148 INFO:tasks.workunit.client.0.smithi157.stdout: "ever_enabled_multiple": true 2024-09-18T11:55:09.148 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.148 INFO:tasks.workunit.client.0.smithi157.stdout: "standbys": [], 2024-09-18T11:55:09.148 INFO:tasks.workunit.client.0.smithi157.stdout: "filesystems": [] 2024-09-18T11:55:09.148 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.148 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_first_committed": 1, 2024-09-18T11:55:09.148 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_last_committed": 1, 2024-09-18T11:55:09.148 INFO:tasks.workunit.client.0.smithi157.stdout: "auth": { 2024-09-18T11:55:09.148 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-18T11:55:09.148 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 3, 2024-09-18T11:55:09.148 INFO:tasks.workunit.client.0.smithi157.stdout: "num_secrets": 1 2024-09-18T11:55:09.148 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.148 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 4, 2024-09-18T11:55:09.149 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_active": 4, 2024-09-18T11:55:09.149 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_unknown": 0, 2024-09-18T11:55:09.149 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osd": 1, 2024-09-18T11:55:09.149 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_sum": { 2024-09-18T11:55:09.149 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-18T11:55:09.149 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 19, 2024-09-18T11:55:09.149 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 2, 2024-09-18T11:55:09.149 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 0, 2024-09-18T11:55:09.149 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 4, 2024-09-18T11:55:09.149 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T11:55:09.149 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-18T11:55:09.149 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 2, 2024-09-18T11:55:09.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-18T11:55:09.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-18T11:55:09.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 2, 2024-09-18T11:55:09.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-18T11:55:09.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-18T11:55:09.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-18T11:55:09.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 2, 2024-09-18T11:55:09.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 2, 2024-09-18T11:55:09.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-18T11:55:09.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T11:55:09.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-18T11:55:09.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-18T11:55:09.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-18T11:55:09.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-18T11:55:09.151 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-18T11:55:09.151 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T11:55:09.151 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T11:55:09.151 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-18T11:55:09.151 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-18T11:55:09.151 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-18T11:55:09.151 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-18T11:55:09.151 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-18T11:55:09.151 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-18T11:55:09.151 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-18T11:55:09.151 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-18T11:55:09.151 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-18T11:55:09.152 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-18T11:55:09.152 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-18T11:55:09.152 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-18T11:55:09.152 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-18T11:55:09.152 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-18T11:55:09.152 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-18T11:55:09.152 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-18T11:55:09.152 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.152 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-18T11:55:09.152 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-18T11:55:09.152 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-18T11:55:09.152 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-18T11:55:09.152 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 0, 2024-09-18T11:55:09.153 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 0, 2024-09-18T11:55:09.153 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-18T11:55:09.153 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-18T11:55:09.153 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-18T11:55:09.153 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-18T11:55:09.153 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-18T11:55:09.153 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.153 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 3, 2024-09-18T11:55:09.153 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 3, 2024-09-18T11:55:09.153 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 4, 2024-09-18T11:55:09.153 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 4, 2024-09-18T11:55:09.153 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 0 2024-09-18T11:55:09.153 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.153 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum": { 2024-09-18T11:55:09.153 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-18T11:55:09.154 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-18T11:55:09.154 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 4, 2024-09-18T11:55:09.154 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 1, 2024-09-18T11:55:09.154 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 1, 2024-09-18T11:55:09.154 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 1, 2024-09-18T11:55:09.154 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 104857600, 2024-09-18T11:55:09.154 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 297296, 2024-09-18T11:55:09.154 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 116, 2024-09-18T11:55:09.154 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 1, 2024-09-18T11:55:09.154 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 297150, 2024-09-18T11:55:09.154 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 104560304, 2024-09-18T11:55:09.154 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-18T11:55:09.154 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 107374182400, 2024-09-18T11:55:09.154 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 107069751296, 2024-09-18T11:55:09.154 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-18T11:55:09.155 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 118784, 2024-09-18T11:55:09.155 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 24598, 2024-09-18T11:55:09.155 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-18T11:55:09.155 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-18T11:55:09.155 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-18T11:55:09.155 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 1704, 2024-09-18T11:55:09.155 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 304281944 2024-09-18T11:55:09.155 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.155 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-18T11:55:09.155 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-18T11:55:09.155 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-18T11:55:09.155 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 0, 2024-09-18T11:55:09.155 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-18T11:55:09.155 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-18T11:55:09.155 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-18T11:55:09.156 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.156 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-18T11:55:09.156 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 0, 2024-09-18T11:55:09.156 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 0, 2024-09-18T11:55:09.156 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 0, 2024-09-18T11:55:09.156 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 0 2024-09-18T11:55:09.156 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.156 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-18T11:55:09.156 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-18T11:55:09.156 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.156 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum_by_class": { 2024-09-18T11:55:09.156 INFO:tasks.workunit.client.0.smithi157.stdout: "hdd": { 2024-09-18T11:55:09.156 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-18T11:55:09.157 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-18T11:55:09.157 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 4, 2024-09-18T11:55:09.157 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 1, 2024-09-18T11:55:09.157 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 1, 2024-09-18T11:55:09.157 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 1, 2024-09-18T11:55:09.157 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 104857600, 2024-09-18T11:55:09.157 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 297296, 2024-09-18T11:55:09.157 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 116, 2024-09-18T11:55:09.157 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 1, 2024-09-18T11:55:09.157 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 297150, 2024-09-18T11:55:09.157 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 104560304, 2024-09-18T11:55:09.157 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-18T11:55:09.157 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 107374182400, 2024-09-18T11:55:09.158 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 107069751296, 2024-09-18T11:55:09.158 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-18T11:55:09.158 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 118784, 2024-09-18T11:55:09.158 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 24598, 2024-09-18T11:55:09.158 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-18T11:55:09.158 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-18T11:55:09.158 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-18T11:55:09.158 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 1704, 2024-09-18T11:55:09.158 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 304281944 2024-09-18T11:55:09.158 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.158 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-18T11:55:09.158 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-18T11:55:09.158 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-18T11:55:09.158 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 0, 2024-09-18T11:55:09.159 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-18T11:55:09.159 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-18T11:55:09.159 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-18T11:55:09.159 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.159 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-18T11:55:09.159 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 0, 2024-09-18T11:55:09.159 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 0, 2024-09-18T11:55:09.159 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 0, 2024-09-18T11:55:09.159 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 0 2024-09-18T11:55:09.159 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.159 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-18T11:55:09.159 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-18T11:55:09.159 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.159 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.160 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_stats": [ 2024-09-18T11:55:09.160 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.160 INFO:tasks.workunit.client.0.smithi157.stdout: "poolid": 1, 2024-09-18T11:55:09.160 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 4, 2024-09-18T11:55:09.160 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-18T11:55:09.160 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 19, 2024-09-18T11:55:09.160 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 2, 2024-09-18T11:55:09.160 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 0, 2024-09-18T11:55:09.160 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 4, 2024-09-18T11:55:09.160 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T11:55:09.160 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-18T11:55:09.160 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 2, 2024-09-18T11:55:09.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-18T11:55:09.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-18T11:55:09.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 2, 2024-09-18T11:55:09.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-18T11:55:09.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-18T11:55:09.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-18T11:55:09.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 2, 2024-09-18T11:55:09.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 2, 2024-09-18T11:55:09.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-18T11:55:09.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T11:55:09.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-18T11:55:09.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-18T11:55:09.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-18T11:55:09.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-18T11:55:09.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-18T11:55:09.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T11:55:09.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T11:55:09.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-18T11:55:09.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-18T11:55:09.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-18T11:55:09.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-18T11:55:09.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-18T11:55:09.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-18T11:55:09.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-18T11:55:09.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-18T11:55:09.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-18T11:55:09.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-18T11:55:09.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-18T11:55:09.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-18T11:55:09.163 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-18T11:55:09.163 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-18T11:55:09.163 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-18T11:55:09.163 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-18T11:55:09.163 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.163 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-18T11:55:09.163 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-18T11:55:09.163 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-18T11:55:09.163 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-18T11:55:09.163 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 4096, 2024-09-18T11:55:09.163 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 19, 2024-09-18T11:55:09.163 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-18T11:55:09.163 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-18T11:55:09.163 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-18T11:55:09.163 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-18T11:55:09.164 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-18T11:55:09.164 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.164 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 3, 2024-09-18T11:55:09.164 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 3, 2024-09-18T11:55:09.164 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 4, 2024-09-18T11:55:09.164 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 4, 2024-09-18T11:55:09.164 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 1 2024-09-18T11:55:09.164 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.164 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.164 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_stats": [ 2024-09-18T11:55:09.164 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.164 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-18T11:55:09.164 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 21474836482 2024-09-18T11:55:09.164 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.165 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.165 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_state": [ 2024-09-18T11:55:09.165 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.165 INFO:tasks.workunit.client.0.smithi157.stdout: "state": "active+undersized+degraded", 2024-09-18T11:55:09.165 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 2 2024-09-18T11:55:09.165 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.165 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.165 INFO:tasks.workunit.client.0.smithi157.stdout: "state": "active+undersized", 2024-09-18T11:55:09.165 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 2 2024-09-18T11:55:09.165 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.165 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.165 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_osd": [ 2024-09-18T11:55:09.165 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.165 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-18T11:55:09.166 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 4, 2024-09-18T11:55:09.166 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 4, 2024-09-18T11:55:09.166 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-18T11:55:09.166 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.166 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.166 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": [ 2024-09-18T11:55:09.166 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:55:09.166 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-18T11:55:09.166 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": {} 2024-09-18T11:55:09.166 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.166 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:55:09.166 INFO:tasks.workunit.client.0.smithi157.stdout: "servicemap": { 2024-09-18T11:55:09.166 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-18T11:55:09.166 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-18T11:54:24.133444+0000", 2024-09-18T11:55:09.167 INFO:tasks.workunit.client.0.smithi157.stdout: "services": {} 2024-09-18T11:55:09.167 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:55:09.167 INFO:tasks.workunit.client.0.smithi157.stdout: "paxos": { 2024-09-18T11:55:09.167 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-18T11:55:09.167 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 105, 2024-09-18T11:55:09.167 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pn": 0, 2024-09-18T11:55:09.167 INFO:tasks.workunit.client.0.smithi157.stdout: "accepted_pn": 0 2024-09-18T11:55:09.167 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:55:09.167 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:55:09.170 INFO:tasks.workunit.client.0.smithi157.stderr:report 3945138977 2024-09-18T11:55:09.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-18T11:55:09.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: test_wait_for_clean: run_osd td/ceph-helpers 1 2024-09-18T11:55:09.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:55:09.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:55:09.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:55:09.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:55:09.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-18T11:55:09.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:55:09.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:55:09.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:55:09.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:55:09.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-18T11:55:09.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-18T11:55:09.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:55:09.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:55:09.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:55:09.172 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:55:09.172 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:55:09.172 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:55:09.172 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:55:09.172 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:55:09.172 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:55:09.172 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:55:09.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:55:09.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:55:09.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:55:09.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:55:09.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:55:09.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:55:09.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:55:09.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:55:09.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:55:09.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:55:09.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:55:09.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:55:09.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-18T11:55:09.173 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:55:09.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8ed0b323-8c36-4029-b528-bdd5f960d15f 2024-09-18T11:55:09.174 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 8ed0b323-8c36-4029-b528-bdd5f960d15f 2024-09-18T11:55:09.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8ed0b323-8c36-4029-b528-bdd5f960d15f' 2024-09-18T11:55:09.174 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:55:09.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCdv+pmn23eBxAA2RwfI0hGqMnaFXV1S4813A== 2024-09-18T11:55:09.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCdv+pmn23eBxAA2RwfI0hGqMnaFXV1S4813A=="}' 2024-09-18T11:55:09.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8ed0b323-8c36-4029-b528-bdd5f960d15f -i td/ceph-helpers/1/new.json 2024-09-18T11:55:09.614 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:55:09.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-18T11:55:09.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdv+pmn23eBxAA2RwfI0hGqMnaFXV1S4813A== --osd-uuid 8ed0b323-8c36-4029-b528-bdd5f960d15f 2024-09-18T11:55:09.650 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:09.648+0000 7f2bcd216440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:55:09.651 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:09.650+0000 7f2bcd216440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:55:09.653 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:09.651+0000 7f2bcd216440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:55:09.653 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:09.651+0000 7f2bcd216440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:55:09.653 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:09.651+0000 7f2bcd216440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:55:09.653 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:09.651+0000 7f2bcd216440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:55:09.704 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:09.703+0000 7f2bcd216440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-18T11:55:15.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-18T11:55:15.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:55:15.696 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T11:55:15.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:55:15.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:55:16.181 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T11:55:16.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:55:16.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:55:16.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:55:16.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:55:16.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:55:16.204 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:16.202+0000 7f29e2e0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:55:16.205 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:16.204+0000 7f29e2e0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:55:16.206 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:16.205+0000 7f29e2e0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:55:16.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:55:16.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:55:16.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:55:16.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:55:16.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:55:16.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:16.538 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:55:16.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:55:16.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:16.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:16.723 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:16.722+0000 7f29e2e0a440 -1 Falling back to public interface 2024-09-18T11:55:16.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:17.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:17.925 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:55:17.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:17.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:55:17.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:17.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:18.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:19.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:19.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:19.285 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:55:19.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:55:19.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:19.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:19.296 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:19.295+0000 7f29e2e0a440 -1 osd.1 0 log_to_monitors true 2024-09-18T11:55:19.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:20.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:20.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:20.791 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:55:20.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:55:20.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:20.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:21.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:22.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:22.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:22.228 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:55:22.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:55:22.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:22.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:22.614 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:23.615 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:23.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:23.616 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:55:23.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:55:23.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:23.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:24.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:25.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:25.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:25.107 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:55:25.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:55:25.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:25.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:25.500 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:26.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:26.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:26.502 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:55:26.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:55:26.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:26.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:26.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:27.865 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:27.865 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:27.865 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:55:27.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:55:27.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:27.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:28.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:29.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:29.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:29.274 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:55:29.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:55:29.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:29.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:29.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:30.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:30.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:30.668 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:55:30.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:55:30.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:30.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:31.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:32.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:32.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:32.075 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:55:32.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:55:32.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:32.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:32.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:33.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:33.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:33.809 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:55:33.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:55:33.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:33.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:34.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:35.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:35.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:35.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:55:35.521 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:55:35.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:35.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:36.289 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:37.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:37.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:37.291 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:55:37.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:55:37.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:37.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:38.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:39.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:39.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:39.092 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:55:39.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:55:39.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:39.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:39.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:40.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:40.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:40.887 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:55:40.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:55:40.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:40.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:41.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:42.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:42.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:42.697 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:55:42.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:55:42.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:42.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:55:43.279 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 17 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/151063,v1:127.0.0.1:6811/151063] [v2:127.0.0.1:6812/151063,v1:127.0.0.1:6813/151063] exists,up 8ed0b323-8c36-4029-b528-bdd5f960d15f 2024-09-18T11:55:43.280 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:55:43.280 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:55:43.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:55:43.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1686: test_wait_for_clean: wait_for_clean 2024-09-18T11:55:43.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:55:43.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:55:43.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:55:43.281 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:55:43.281 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:55:43.281 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:55:43.281 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:55:43.281 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:55:43.281 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:55:43.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T11:55:43.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:55:43.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:55:43.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:55:43.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:55:43.367 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:55:43.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:55:43.941 INFO:tasks.workunit.client.0.smithi157.stderr:1' 2024-09-18T11:55:43.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:55:43.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:55:43.941 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:55:44.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836491 2024-09-18T11:55:44.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836491 2024-09-18T11:55:44.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836491' 2024-09-18T11:55:44.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:55:44.049 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:55:44.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=73014444036 2024-09-18T11:55:44.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 73014444036 2024-09-18T11:55:44.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836491 1-73014444036' 2024-09-18T11:55:44.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:55:44.156 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836491 2024-09-18T11:55:44.156 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:55:44.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:55:44.157 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836491 2024-09-18T11:55:44.157 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:55:44.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836491 2024-09-18T11:55:44.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-09-18T11:55:44.158 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836491 2024-09-18T11:55:44.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:55:44.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836491 -lt 21474836491 2024-09-18T11:55:44.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:55:44.558 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-73014444036 2024-09-18T11:55:44.558 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:55:44.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:55:44.559 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-73014444036 2024-09-18T11:55:44.560 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:55:44.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=73014444036 2024-09-18T11:55:44.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 73014444036' 2024-09-18T11:55:44.561 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 73014444036 2024-09-18T11:55:44.561 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:55:44.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 73014444036 -lt 73014444036 2024-09-18T11:55:44.953 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:55:44.953 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:55:44.953 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:55:45.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:55:45.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:55:45.441 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:55:45.441 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:55:45.441 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:55:45.441 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:55:45.441 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:55:45.441 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:55:45.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:55:45.874 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:55:45.874 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:55:45.874 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:55:46.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T11:55:46.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:55:46.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:55:46.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: test_wait_for_clean: teardown td/ceph-helpers 2024-09-18T11:55:46.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:55:46.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:55:46.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:55:46.341 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:55:46.342 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:55:46.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:55:46.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:55:46.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:55:46.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:55:46.447 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:55:46.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:55:46.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:55:46.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:55:46.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:55:46.450 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:55:46.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:55:46.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:55:46.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:55:46.451 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:55:46.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:55:46.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:55:46.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:55:46.462 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:55:46.462 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:55:46.462 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:55:46.463 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:55:46.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:55:46.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:55:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:55:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_wait_for_health_ok td/ceph-helpers 2024-09-18T11:55:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1822: test_wait_for_health_ok: local dir=td/ceph-helpers 2024-09-18T11:55:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1824: test_wait_for_health_ok: setup td/ceph-helpers 2024-09-18T11:55:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:55:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:55:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:55:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:55:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:55:46.465 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:55:46.465 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:55:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:55:46.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:55:46.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:55:46.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:55:46.466 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:55:46.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:55:46.467 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:55:46.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:55:46.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:55:46.468 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:55:46.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:55:46.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:55:46.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:55:46.469 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:55:46.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:55:46.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:55:46.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:55:46.472 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:55:46.472 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:55:46.472 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:55:46.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:55:46.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:55:46.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:55:46.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:55:46.474 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:55:46.474 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:55:46.474 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:55:46.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:55:46.476 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:55:46.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:55:46.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:55:46.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:55:46.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1825: test_wait_for_health_ok: run_mon td/ceph-helpers a --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-09-18T11:55:46.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:55:46.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:55:46.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:55:46.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:55:46.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:55:46.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-09-18T11:55:46.825 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:55:46.826 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:55:46.826 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:55:46.826 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:55:46.826 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:55:46.826 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:55:46.826 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:55:46.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-09-18T11:55:47.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:55:47.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:55:47.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:55:47.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:55:47.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:55:47.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:55:47.094 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:55:47.094 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:55:47.095 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:55:47.095 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:55:47.095 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:55:47.096 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:55:47.096 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:55:47.096 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:55:47.096 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:55:47.169 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:55:47.169 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:55:47.169 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:55:47.169 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:55:47.169 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:55:47.170 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:55:47.170 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:55:47.170 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:55:47.170 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:55:47.170 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:55:47.170 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:55:47.170 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:55:47.171 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:55:47.171 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:55:47.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1826: test_wait_for_health_ok: run_mgr td/ceph-helpers x --mon_pg_warn_min_per_osd=0 2024-09-18T11:55:47.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:55:47.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:55:47.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:55:47.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:55:47.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:55:47.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:55:48.080 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:55:48.080 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:55:48.081 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:55:48.081 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:55:48.081 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:55:48.081 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:55:48.081 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:55:48.081 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:55:48.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_pg_warn_min_per_osd=0 2024-09-18T11:55:48.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1828: test_wait_for_health_ok: run_osd td/ceph-helpers 0 2024-09-18T11:55:48.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:55:48.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:55:48.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:55:48.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:55:48.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:55:48.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:55:48.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:55:48.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:55:48.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:55:48.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:55:48.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:55:48.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:55:48.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:55:48.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:55:48.109 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:55:48.109 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:55:48.109 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:55:48.109 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:55:48.109 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:55:48.109 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:55:48.109 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:55:48.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:55:48.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:55:48.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:55:48.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:55:48.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:55:48.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:55:48.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:55:48.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:55:48.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:55:48.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:55:48.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:55:48.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:55:48.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:55:48.111 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:55:48.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9f3eac03-288d-4961-a651-5b82830ebf98 2024-09-18T11:55:48.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9f3eac03-288d-4961-a651-5b82830ebf98' 2024-09-18T11:55:48.112 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 9f3eac03-288d-4961-a651-5b82830ebf98 2024-09-18T11:55:48.112 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:55:48.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDEv+pmH+hQBxAAJ0fFzIRngrblpCgwhVBuHA== 2024-09-18T11:55:48.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDEv+pmH+hQBxAAJ0fFzIRngrblpCgwhVBuHA=="}' 2024-09-18T11:55:48.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9f3eac03-288d-4961-a651-5b82830ebf98 -i td/ceph-helpers/0/new.json 2024-09-18T11:55:48.432 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:55:48.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:55:48.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDEv+pmH+hQBxAAJ0fFzIRngrblpCgwhVBuHA== --osd-uuid 9f3eac03-288d-4961-a651-5b82830ebf98 2024-09-18T11:55:48.466 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:48.465+0000 7fb81ba37440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:55:48.478 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:48.477+0000 7fb81ba37440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:55:48.479 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:48.478+0000 7fb81ba37440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:55:48.479 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:48.478+0000 7fb81ba37440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:55:48.480 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:48.478+0000 7fb81ba37440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:55:48.480 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:48.478+0000 7fb81ba37440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:55:48.518 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:48.517+0000 7fb81ba37440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:55:54.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:55:54.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:55:54.092 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:55:54.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:55:54.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:55:54.585 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:55:54.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:55:54.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:55:54.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:55:54.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:55:54.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:55:54.607 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:54.606+0000 7f2b76420440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:55:54.609 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:54.608+0000 7f2b76420440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:55:54.610 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:54.609+0000 7f2b76420440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:55:54.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:55:54.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:55:54.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:55:54.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:55:54.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:55:54.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:54.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:55:54.948 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:55:54.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:54.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:55:55.147 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:55.145+0000 7f2b76420440 -1 Falling back to public interface 2024-09-18T11:55:55.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:56.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:56.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:56.305 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:55:56.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:55:56.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:56.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:55:56.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:57.650 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:55:57.649+0000 7f2b76420440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:55:57.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:57.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:57.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:55:57.675 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:55:57.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:57.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:55:58.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:55:59.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:55:59.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:55:59.085 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:55:59.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:55:59.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:55:59.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:55:59.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:00.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:00.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:00.507 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:56:00.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:56:00.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:00.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:56:00.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:01.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:01.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:01.905 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:56:01.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:56:01.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:01.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:56:02.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:03.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:03.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:03.402 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:56:03.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:56:03.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:03.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:56:03.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:04.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:04.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:04.766 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:56:04.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:56:04.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:04.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:56:05.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:06.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:06.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:06.163 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:56:06.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:56:06.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:06.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:56:06.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:07.548 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:07.548 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:07.548 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:56:07.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:56:07.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:07.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:56:07.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:08.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:08.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:08.945 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:56:08.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:56:08.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:08.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:56:09.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:10.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:10.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:10.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:56:10.328 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:56:10.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:10.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:56:10.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:11.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:11.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:11.792 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:56:11.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:56:11.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:11.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:56:12.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:13.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:13.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:13.577 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:56:13.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:56:13.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:13.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:56:14.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:15.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:15.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:15.062 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:56:15.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:56:15.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:15.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:56:15.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:16.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:16.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:16.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:56:16.782 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:56:16.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:16.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:56:17.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:18.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:18.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:18.470 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:56:18.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:56:18.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:18.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:56:19.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:20.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:20.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:20.294 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:56:20.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:56:20.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:20.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:56:20.658 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/153344,v1:127.0.0.1:6803/153344] [v2:127.0.0.1:6804/153344,v1:127.0.0.1:6805/153344] exists,up 9f3eac03-288d-4961-a651-5b82830ebf98 2024-09-18T11:56:20.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:56:20.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:56:20.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:56:20.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1829: test_wait_for_health_ok: run_osd td/ceph-helpers 1 2024-09-18T11:56:20.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:56:20.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:56:20.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:56:20.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:56:20.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-18T11:56:20.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:56:20.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:56:20.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:56:20.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:56:20.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-18T11:56:20.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-18T11:56:20.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:56:20.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:56:20.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:56:20.663 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:56:20.663 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:56:20.663 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:56:20.663 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:56:20.663 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:56:20.664 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:56:20.664 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:56:20.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:56:20.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:56:20.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:56:20.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:56:20.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:56:20.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:56:20.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:56:20.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:56:20.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:56:20.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:56:20.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:56:20.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:56:20.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-18T11:56:20.665 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:56:20.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c2967e7b-e2ba-4cec-9b09-ca92c9e6e5d0 2024-09-18T11:56:20.665 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 c2967e7b-e2ba-4cec-9b09-ca92c9e6e5d0 2024-09-18T11:56:20.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c2967e7b-e2ba-4cec-9b09-ca92c9e6e5d0' 2024-09-18T11:56:20.665 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:56:20.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkv+pmc7A/KBAAkHfqeqZ8ZRGgO7H0Sii04Q== 2024-09-18T11:56:20.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkv+pmc7A/KBAAkHfqeqZ8ZRGgO7H0Sii04Q=="}' 2024-09-18T11:56:20.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c2967e7b-e2ba-4cec-9b09-ca92c9e6e5d0 -i td/ceph-helpers/1/new.json 2024-09-18T11:56:21.190 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:56:21.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-18T11:56:21.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkv+pmc7A/KBAAkHfqeqZ8ZRGgO7H0Sii04Q== --osd-uuid c2967e7b-e2ba-4cec-9b09-ca92c9e6e5d0 2024-09-18T11:56:21.227 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:21.225+0000 7f684c28d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:56:21.244 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:21.243+0000 7f684c28d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:56:21.245 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:21.244+0000 7f684c28d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:56:21.245 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:21.244+0000 7f684c28d440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:56:21.246 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:21.244+0000 7f684c28d440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:56:21.246 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:21.244+0000 7f684c28d440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:56:21.284 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:21.282+0000 7f684c28d440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-18T11:56:27.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-18T11:56:27.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:56:27.514 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T11:56:27.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:56:27.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:56:28.000 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T11:56:28.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:56:28.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:56:28.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:56:28.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:56:28.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:56:28.023 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:28.021+0000 7fbc82814440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:56:28.024 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:28.023+0000 7fbc82814440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:56:28.026 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:28.024+0000 7fbc82814440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:56:28.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:56:28.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:56:28.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:56:28.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:56:28.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:56:28.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:28.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:56:28.392 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:56:28.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:28.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:28.543 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:28.542+0000 7fbc82814440 -1 Falling back to public interface 2024-09-18T11:56:28.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:29.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:29.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:29.746 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:56:29.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:56:29.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:29.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:30.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:31.076 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:31.074+0000 7fbc82814440 -1 osd.1 0 log_to_monitors true 2024-09-18T11:56:31.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:31.104 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:56:31.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:31.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:56:31.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:31.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:31.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:32.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:32.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:32.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:56:32.746 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:56:32.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:32.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:33.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:34.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:34.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:34.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:56:34.167 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:56:34.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:34.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:34.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:35.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:35.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:35.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:56:35.619 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:56:35.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:35.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:36.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:37.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:37.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:37.161 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:56:37.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:56:37.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:37.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:37.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:38.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:38.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:38.528 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:56:38.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:56:38.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:38.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:38.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:39.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:39.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:39.944 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:56:39.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:56:39.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:39.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:40.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:41.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:41.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:41.434 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:56:41.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:56:41.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:41.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:41.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:42.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:42.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:42.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:56:42.943 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:56:42.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:42.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:43.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:44.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:44.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:44.441 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:56:44.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:56:44.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:44.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:45.247 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:46.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:46.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:46.249 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:56:46.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:56:46.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:46.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:47.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:48.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:48.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:48.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:56:48.305 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:56:48.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:48.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:49.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:50.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:50.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:50.132 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:56:50.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:56:50.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:50.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:50.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:51.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:51.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:51.919 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:56:51.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:56:51.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:51.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:52.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:53.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:53.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:53.774 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:56:53.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:56:53.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:53.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:54.339 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:56:55.340 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:56:55.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:56:55.341 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:56:55.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:56:55.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:56:55.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:56:55.696 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/155099,v1:127.0.0.1:6811/155099] [v2:127.0.0.1:6812/155099,v1:127.0.0.1:6813/155099] exists,up c2967e7b-e2ba-4cec-9b09-ca92c9e6e5d0 2024-09-18T11:56:55.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:56:55.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:56:55.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:56:55.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: test_wait_for_health_ok: run_osd td/ceph-helpers 2 2024-09-18T11:56:55.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:56:55.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:56:55.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:56:55.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:56:55.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-09-18T11:56:55.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:56:55.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:56:55.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:56:55.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:56:55.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-09-18T11:56:55.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-09-18T11:56:55.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:56:55.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:56:55.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:56:55.702 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:56:55.702 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:56:55.702 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:56:55.702 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:56:55.702 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:56:55.702 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:56:55.702 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:56:55.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:56:55.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:56:55.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:56:55.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:56:55.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:56:55.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:56:55.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:56:55.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:56:55.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:56:55.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:56:55.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:56:55.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:56:55.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-09-18T11:56:55.703 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:56:55.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c3545f85-04aa-496e-b713-7a5d738c5473 2024-09-18T11:56:55.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c3545f85-04aa-496e-b713-7a5d738c5473' 2024-09-18T11:56:55.704 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 c3545f85-04aa-496e-b713-7a5d738c5473 2024-09-18T11:56:55.704 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:56:55.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAHwOpmKgaJKhAADedp/88JvIEVXaLWhu+okg== 2024-09-18T11:56:55.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAHwOpmKgaJKhAADedp/88JvIEVXaLWhu+okg=="}' 2024-09-18T11:56:55.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c3545f85-04aa-496e-b713-7a5d738c5473 -i td/ceph-helpers/2/new.json 2024-09-18T11:56:56.185 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:56:56.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-09-18T11:56:56.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHwOpmKgaJKhAADedp/88JvIEVXaLWhu+okg== --osd-uuid c3545f85-04aa-496e-b713-7a5d738c5473 2024-09-18T11:56:56.220 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:56.219+0000 7f0c66c09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:56:56.222 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:56.220+0000 7f0c66c09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:56:56.223 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:56.221+0000 7f0c66c09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:56:56.223 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:56.222+0000 7f0c66c09440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-18T11:56:56.223 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:56.222+0000 7f0c66c09440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-18T11:56:56.224 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:56.222+0000 7f0c66c09440 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-18T11:56:56.300 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:56:56.298+0000 7f0c66c09440 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-09-18T11:57:02.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-09-18T11:57:02.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:57:02.148 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-18T11:57:02.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:57:02.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:57:02.661 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-18T11:57:02.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:57:02.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:57:02.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:57:02.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:57:02.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:57:02.684 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:02.682+0000 7f8052a35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:02.686 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:02.684+0000 7f8052a35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:02.687 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:02.685+0000 7f8052a35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:03.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:57:03.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:57:03.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:57:03.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:57:03.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:57:03.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:03.014 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:57:03.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:57:03.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:03.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:03.199 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:03.198+0000 7f8052a35440 -1 Falling back to public interface 2024-09-18T11:57:03.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:04.381 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:04.381 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:57:04.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:04.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:57:04.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:04.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:04.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:05.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:05.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:05.732 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:57:05.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:57:05.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:05.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:05.829 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:05.827+0000 7f8052a35440 -1 osd.2 0 log_to_monitors true 2024-09-18T11:57:06.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:07.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:07.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:07.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:57:07.147 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:57:07.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:07.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:07.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:08.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:08.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:08.514 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:57:08.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:57:08.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:08.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:09.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:10.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:10.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:10.042 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:57:10.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:57:10.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:10.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:10.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:11.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:11.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:11.557 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:57:11.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:57:11.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:11.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:12.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:13.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:13.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:13.003 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:57:13.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:57:13.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:13.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:13.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:14.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:14.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:14.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:57:14.379 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:57:14.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:14.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:14.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:15.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:15.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:15.853 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:57:15.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:57:15.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:15.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:16.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:17.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:17.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:17.276 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:57:17.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:57:17.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:17.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:17.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:18.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:18.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:18.750 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:57:18.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:57:18.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:18.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:19.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:20.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:20.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:20.122 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:57:20.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:57:20.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:20.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:20.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:21.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:21.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:21.881 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:57:21.881 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:57:21.881 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:21.881 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:22.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:23.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:23.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:23.944 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:57:23.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:57:23.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:23.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:24.920 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:25.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:25.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:25.922 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:57:25.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:57:25.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:25.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:26.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:27.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:27.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:27.742 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:57:27.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:57:27.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:27.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:28.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:29.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:29.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:29.601 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:57:29.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:57:29.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:29.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:30.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:31.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:31.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:31.278 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T11:57:31.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:57:31.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:31.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:31.634 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/156853,v1:127.0.0.1:6819/156853] [v2:127.0.0.1:6820/156853,v1:127.0.0.1:6821/156853] exists,up c3545f85-04aa-496e-b713-7a5d738c5473 2024-09-18T11:57:31.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:57:31.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:57:31.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:57:31.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1831: test_wait_for_health_ok: kill_daemons td/ceph-helpers TERM osd 2024-09-18T11:57:31.635 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:57:31.636 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:57:31.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:57:31.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:57:31.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:57:31.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:57:31.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1832: test_wait_for_health_ok: ceph osd down 0 2024-09-18T11:57:32.304 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already down. 2024-09-18T11:57:32.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: test_wait_for_health_ok: TIMEOUT=1 2024-09-18T11:57:32.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: test_wait_for_health_ok: wait_for_health_ok 2024-09-18T11:57:32.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: wait_for_health_ok: wait_for_health HEALTH_OK 2024-09-18T11:57:32.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_health: local grepstr=HEALTH_OK 2024-09-18T11:57:32.316 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: get_timeout_delays 1 .1 2024-09-18T11:57:32.317 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:57:32.317 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:57:32.317 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:57:32.317 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:57:32.317 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:57:32.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: delays=('0.1' '0.2' '0.4' '0.3') 2024-09-18T11:57:32.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: local -a delays 2024-09-18T11:57:32.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: wait_for_health: local -i loop=0 2024-09-18T11:57:32.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-09-18T11:57:32.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-09-18T11:57:32.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 0 >= 4 )) 2024-09-18T11:57:32.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: wait_for_health: sleep 0.1 2024-09-18T11:57:32.912 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health: loop+=1 2024-09-18T11:57:32.912 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-09-18T11:57:32.912 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-09-18T11:57:33.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 1 >= 4 )) 2024-09-18T11:57:33.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: wait_for_health: sleep 0.2 2024-09-18T11:57:33.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health: loop+=1 2024-09-18T11:57:33.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-09-18T11:57:33.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-09-18T11:57:34.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 2 >= 4 )) 2024-09-18T11:57:34.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: wait_for_health: sleep 0.4 2024-09-18T11:57:34.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health: loop+=1 2024-09-18T11:57:34.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-09-18T11:57:34.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-09-18T11:57:34.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 3 >= 4 )) 2024-09-18T11:57:34.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: wait_for_health: sleep 0.3 2024-09-18T11:57:35.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health: loop+=1 2024-09-18T11:57:35.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-09-18T11:57:35.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-09-18T11:57:35.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 4 >= 4 )) 2024-09-18T11:57:35.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1803: wait_for_health: ceph health detail 2024-09-18T11:57:36.218 INFO:tasks.workunit.client.0.smithi157.stdout:HEALTH_WARN 3 osds down; 1 host (3 osds) down; 1 root (3 osds) down 2024-09-18T11:57:36.218 INFO:tasks.workunit.client.0.smithi157.stdout:[WRN] OSD_DOWN: 3 osds down 2024-09-18T11:57:36.218 INFO:tasks.workunit.client.0.smithi157.stdout: osd.0 (root=default,host=smithi157) is down 2024-09-18T11:57:36.218 INFO:tasks.workunit.client.0.smithi157.stdout: osd.1 (root=default,host=smithi157) is down 2024-09-18T11:57:36.218 INFO:tasks.workunit.client.0.smithi157.stdout: osd.2 (root=default,host=smithi157) is down 2024-09-18T11:57:36.218 INFO:tasks.workunit.client.0.smithi157.stdout:[WRN] OSD_HOST_DOWN: 1 host (3 osds) down 2024-09-18T11:57:36.219 INFO:tasks.workunit.client.0.smithi157.stdout: host smithi157 (root=default) (3 osds) is down 2024-09-18T11:57:36.219 INFO:tasks.workunit.client.0.smithi157.stdout:[WRN] OSD_ROOT_DOWN: 1 root (3 osds) down 2024-09-18T11:57:36.219 INFO:tasks.workunit.client.0.smithi157.stdout: root default (3 osds) is down 2024-09-18T11:57:36.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1804: wait_for_health: return 1 2024-09-18T11:57:36.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: wait_for_health_ok: return 1 2024-09-18T11:57:36.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: test_wait_for_health_ok: activate_osd td/ceph-helpers 0 2024-09-18T11:57:36.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-18T11:57:36.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:57:36.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T11:57:36.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:57:36.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:57:36.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:57:36.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:57:36.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:57:36.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:57:36.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:57:36.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:57:36.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:57:36.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:57:36.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:57:36.235 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:57:36.235 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:57:36.235 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:57:36.235 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:57:36.235 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:57:36.235 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:57:36.235 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:57:36.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:57:36.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:57:36.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:57:36.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:57:36.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:57:36.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:57:36.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:57:36.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:57:36.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:57:36.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:57:36.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:57:36.236 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:57:36.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T11:57:36.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:57:36.238 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-18T11:57:36.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T11:57:36.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:57:36.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:57:36.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:57:36.254 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:36.252+0000 7f5304f26440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:36.256 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:36.254+0000 7f5304f26440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:36.257 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:36.255+0000 7f5304f26440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:36.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T11:57:36.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:57:36.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:57:36.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:57:36.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:57:36.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:36.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:57:36.591 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:57:36.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:36.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:57:36.775 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:36.774+0000 7f5304f26440 -1 Falling back to public interface 2024-09-18T11:57:36.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:37.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:37.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:37.942 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:57:37.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:57:37.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:37.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:57:38.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:39.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:39.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:39.305 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:57:39.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:57:39.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:39.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:57:39.554 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:39.553+0000 7f5304f26440 -1 osd.0 14 log_to_monitors true 2024-09-18T11:57:39.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:40.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:40.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:40.664 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:57:40.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:57:40.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:40.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:57:41.015 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [5,14) [v2:127.0.0.1:6802/158015,v1:127.0.0.1:6803/158015] [v2:127.0.0.1:6804/158015,v1:127.0.0.1:6805/158015] exists,up 9f3eac03-288d-4961-a651-5b82830ebf98 2024-09-18T11:57:41.019 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:57:41.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:57:41.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:57:41.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1837: test_wait_for_health_ok: activate_osd td/ceph-helpers 1 2024-09-18T11:57:41.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-18T11:57:41.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:57:41.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T11:57:41.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:57:41.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/1 2024-09-18T11:57:41.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:57:41.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:57:41.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:57:41.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:57:41.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-18T11:57:41.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-18T11:57:41.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:57:41.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:57:41.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:57:41.022 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:57:41.022 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:57:41.022 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:57:41.022 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:57:41.022 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:57:41.022 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:57:41.022 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:57:41.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:57:41.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:57:41.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:57:41.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:57:41.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:57:41.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:57:41.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:57:41.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:57:41.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:57:41.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:57:41.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/1 2024-09-18T11:57:41.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T11:57:41.024 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T11:57:41.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:57:41.025 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/1/whoami 2024-09-18T11:57:41.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T11:57:41.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:57:41.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:57:41.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:57:41.039 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:41.038+0000 7f9c6d041440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:41.076 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:41.075+0000 7f9c6d041440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:41.077 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:41.076+0000 7f9c6d041440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:41.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T11:57:41.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:57:41.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:57:41.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:57:41.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:57:41.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:41.397 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:57:41.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:57:41.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:41.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:57:41.595 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:41.593+0000 7f9c6d041440 -1 Falling back to public interface 2024-09-18T11:57:41.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:42.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:42.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:42.786 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:57:42.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:57:42.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:42.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:57:43.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:44.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:44.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:44.144 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:57:44.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:57:44.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:44.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:57:44.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:44.767 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:44.766+0000 7f9c6d041440 -1 osd.1 16 log_to_monitors true 2024-09-18T11:57:45.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:45.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:45.528 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:57:45.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:57:45.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:45.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:57:45.924 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 21 up_thru 0 down_at 17 last_clean_interval [10,16) [v2:127.0.0.1:6810/158593,v1:127.0.0.1:6811/158593] [v2:127.0.0.1:6812/158593,v1:127.0.0.1:6813/158593] exists,up c2967e7b-e2ba-4cec-9b09-ca92c9e6e5d0 2024-09-18T11:57:45.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:57:45.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:57:45.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:57:45.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: test_wait_for_health_ok: activate_osd td/ceph-helpers 2 2024-09-18T11:57:45.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-18T11:57:45.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:57:45.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T11:57:45.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:57:45.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/2 2024-09-18T11:57:45.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:57:45.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:57:45.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:57:45.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:57:45.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-09-18T11:57:45.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-09-18T11:57:45.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:57:45.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:57:45.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:57:45.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:57:45.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:57:45.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:57:45.930 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:57:45.930 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:57:45.930 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:57:45.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:57:45.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:57:45.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:57:45.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:57:45.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:57:45.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:57:45.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:57:45.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:57:45.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:57:45.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:57:45.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:57:45.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/2 2024-09-18T11:57:45.932 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-18T11:57:45.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T11:57:45.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:57:45.933 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/2/whoami 2024-09-18T11:57:45.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T11:57:45.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:57:45.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:57:45.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:57:45.949 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:45.947+0000 7f678971d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:45.950 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:45.949+0000 7f678971d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:45.951 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:45.950+0000 7f678971d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:46.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T11:57:46.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:57:46.280 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:57:46.280 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:57:46.280 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:57:46.280 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:46.280 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:57:46.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:57:46.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:46.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:46.467 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:46.466+0000 7f678971d440 -1 Falling back to public interface 2024-09-18T11:57:46.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:47.633 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:47.633 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:57:47.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:47.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:57:47.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:47.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:47.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:48.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:48.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:48.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:57:48.997 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:57:48.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:48.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:49.390 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:57:49.463 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:49.462+0000 7f678971d440 -1 osd.2 15 log_to_monitors true 2024-09-18T11:57:50.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:57:50.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:57:50.392 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:57:50.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:57:50.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:57:50.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:57:50.749 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 17 last_clean_interval [0,0) [v2:127.0.0.1:6818/159171,v1:127.0.0.1:6819/159171] [v2:127.0.0.1:6820/159171,v1:127.0.0.1:6821/159171] exists,up c3545f85-04aa-496e-b713-7a5d738c5473 2024-09-18T11:57:50.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:57:50.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:57:50.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:57:50.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: test_wait_for_health_ok: wait_for_health_ok 2024-09-18T11:57:50.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: wait_for_health_ok: wait_for_health HEALTH_OK 2024-09-18T11:57:50.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_health: local grepstr=HEALTH_OK 2024-09-18T11:57:50.754 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: get_timeout_delays 300 .1 2024-09-18T11:57:50.754 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:57:50.754 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:57:50.754 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:57:50.754 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:57:50.754 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:57:50.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2024-09-18T11:57:50.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: local -a delays 2024-09-18T11:57:50.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: wait_for_health: local -i loop=0 2024-09-18T11:57:50.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-09-18T11:57:50.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-09-18T11:57:51.401 INFO:tasks.workunit.client.0.smithi157.stdout:HEALTH_OK 2024-09-18T11:57:51.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: test_wait_for_health_ok: teardown td/ceph-helpers 2024-09-18T11:57:51.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:57:51.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:57:51.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:57:51.402 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:57:51.402 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:57:51.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:57:51.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:57:51.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:57:51.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:57:51.509 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:57:51.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:57:51.510 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:57:51.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:57:51.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:57:51.511 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:57:51.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:57:51.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:57:51.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:57:51.513 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:57:51.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:57:51.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:57:51.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:57:51.527 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:57:51.527 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:57:51.527 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:57:51.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:57:51.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:57:51.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:57:51.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:57:51.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_wait_for_osd td/ceph-helpers 2024-09-18T11:57:51.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:995: test_wait_for_osd: local dir=td/ceph-helpers 2024-09-18T11:57:51.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:996: test_wait_for_osd: setup td/ceph-helpers 2024-09-18T11:57:51.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:57:51.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:57:51.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:57:51.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:57:51.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:57:51.530 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:57:51.531 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:57:51.531 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:57:51.531 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:57:51.531 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:57:51.531 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:57:51.531 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:57:51.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:57:51.532 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:57:51.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:57:51.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:57:51.533 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:57:51.534 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:57:51.534 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:57:51.534 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:57:51.535 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:57:51.535 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:57:51.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:57:51.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:57:51.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:57:51.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:57:51.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:57:51.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:57:51.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:57:51.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:57:51.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:57:51.539 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:57:51.540 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:57:51.540 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:57:51.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:57:51.542 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:57:51.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:57:51.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:57:51.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:57:51.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:997: test_wait_for_osd: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:57:51.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:57:51.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:57:51.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:57:51.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:57:51.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:57:51.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:57:51.884 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:57:51.884 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:57:51.884 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:57:51.884 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:57:51.885 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:57:51.885 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:57:51.885 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:57:51.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T11:57:52.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:57:52.129 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:57:52.129 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:57:52.129 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:57:52.129 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:57:52.129 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:57:52.129 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:57:52.129 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:57:52.129 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:57:52.129 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:57:52.129 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:57:52.129 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:57:52.130 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:57:52.130 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:57:52.130 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:57:52.203 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:57:52.203 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:57:52.203 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:57:52.203 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:57:52.203 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:57:52.203 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:57:52.203 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:57:52.203 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:57:52.203 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:57:52.204 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:57:52.204 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:57:52.204 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:57:52.204 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:57:52.204 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:57:52.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:998: test_wait_for_osd: run_mgr td/ceph-helpers x 2024-09-18T11:57:52.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:57:52.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:57:52.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:57:52.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:57:52.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:57:52.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:57:53.113 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:57:53.113 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:57:53.113 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:57:53.113 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:57:53.113 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:57:53.113 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:57:53.113 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:57:53.114 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:57:53.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:57:53.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:999: test_wait_for_osd: run_osd td/ceph-helpers 0 2024-09-18T11:57:53.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:57:53.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:57:53.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:57:53.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:57:53.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:57:53.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:57:53.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:57:53.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:57:53.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:57:53.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:57:53.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:57:53.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:57:53.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:57:53.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:57:53.142 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:57:53.142 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:57:53.142 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:57:53.142 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:57:53.142 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:57:53.142 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:57:53.142 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:57:53.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:57:53.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:57:53.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:57:53.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:57:53.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:57:53.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:57:53.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:57:53.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:57:53.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:57:53.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:57:53.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:57:53.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:57:53.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:57:53.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:57:53.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=378ba3a6-1d3d-47e1-a984-73f88220d462 2024-09-18T11:57:53.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 378ba3a6-1d3d-47e1-a984-73f88220d462' 2024-09-18T11:57:53.144 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 378ba3a6-1d3d-47e1-a984-73f88220d462 2024-09-18T11:57:53.144 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:57:53.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBwOpmreA6CRAAiCo4hl0W3sQbV5mvqeVQ5w== 2024-09-18T11:57:53.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBwOpmreA6CRAAiCo4hl0W3sQbV5mvqeVQ5w=="}' 2024-09-18T11:57:53.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 378ba3a6-1d3d-47e1-a984-73f88220d462 -i td/ceph-helpers/0/new.json 2024-09-18T11:57:53.482 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:57:53.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:57:53.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBwOpmreA6CRAAiCo4hl0W3sQbV5mvqeVQ5w== --osd-uuid 378ba3a6-1d3d-47e1-a984-73f88220d462 2024-09-18T11:57:53.517 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:53.515+0000 7f9f6fd2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:53.528 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:53.527+0000 7f9f6fd2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:53.529 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:53.528+0000 7f9f6fd2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:53.529 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:53.528+0000 7f9f6fd2f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:57:53.529 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:53.528+0000 7f9f6fd2f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:57:53.530 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:53.528+0000 7f9f6fd2f440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:57:53.568 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:53.567+0000 7f9f6fd2f440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:57:59.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:57:59.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:57:59.256 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:57:59.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:57:59.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:57:59.744 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:57:59.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:57:59.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:57:59.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:57:59.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:57:59.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:57:59.767 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:59.766+0000 7f7e8400c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:59.769 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:59.768+0000 7f7e8400c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:57:59.770 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:57:59.769+0000 7f7e8400c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:58:00.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:58:00.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:58:00.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:58:00.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:58:00.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:58:00.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:00.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:58:00.098 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:58:00.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:00.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:00.286 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:58:00.285+0000 7f7e8400c440 -1 Falling back to public interface 2024-09-18T11:58:00.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:01.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:01.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:01.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:58:01.473 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:58:01.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:01.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:01.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:02.827 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:58:02.825+0000 7f7e8400c440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:58:02.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:02.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:02.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:58:02.832 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:58:02.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:02.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:03.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:04.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:04.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:04.211 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:58:04.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:58:04.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:04.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:04.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:05.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:05.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:05.673 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:58:05.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:58:05.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:05.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:06.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:07.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:07.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:07.050 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:58:07.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:58:07.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:07.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:07.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:08.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:08.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:08.549 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:58:08.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:58:08.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:08.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:08.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:09.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:09.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:09.934 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:58:09.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:58:09.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:09.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:10.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:11.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:11.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:11.323 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:58:11.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:58:11.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:11.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:11.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:12.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:12.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:12.701 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:58:12.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:58:12.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:12.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:13.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:14.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:14.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:14.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:58:14.076 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:58:14.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:14.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:14.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:15.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:15.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:15.526 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:58:15.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:58:15.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:15.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:15.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:16.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:16.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:16.949 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:58:16.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:58:16.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:16.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:17.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:18.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:18.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:18.742 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:58:18.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:58:18.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:18.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:19.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:20.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:20.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:20.179 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:58:20.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:58:20.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:20.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:20.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:21.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:21.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:21.899 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:58:21.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:58:21.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:21.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:22.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:23.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:23.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:23.674 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:58:23.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:58:23.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:23.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:24.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:25.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:25.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:25.738 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:58:25.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:58:25.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:25.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:58:26.094 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/160985,v1:127.0.0.1:6803/160985] [v2:127.0.0.1:6804/160985,v1:127.0.0.1:6805/160985] exists,up 378ba3a6-1d3d-47e1-a984-73f88220d462 2024-09-18T11:58:26.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:58:26.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:58:26.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:58:26.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1000: test_wait_for_osd: run_osd td/ceph-helpers 1 2024-09-18T11:58:26.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:58:26.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:58:26.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:58:26.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:58:26.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-18T11:58:26.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:58:26.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:58:26.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:58:26.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:58:26.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-18T11:58:26.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-18T11:58:26.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:58:26.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:58:26.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:58:26.099 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:58:26.099 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:58:26.099 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:58:26.099 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:58:26.099 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:58:26.099 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:58:26.099 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:58:26.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:58:26.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:58:26.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:58:26.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:58:26.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:58:26.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:58:26.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:58:26.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:58:26.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:58:26.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:58:26.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:58:26.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:58:26.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-18T11:58:26.100 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:58:26.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=781c6a39-5570-465b-9fc0-fa8d3639debb 2024-09-18T11:58:26.101 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 781c6a39-5570-465b-9fc0-fa8d3639debb 2024-09-18T11:58:26.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 781c6a39-5570-465b-9fc0-fa8d3639debb' 2024-09-18T11:58:26.101 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:58:26.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBiwOpmoGidBhAAnIBcklqKQKSr8lkBmUUPiA== 2024-09-18T11:58:26.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBiwOpmoGidBhAAnIBcklqKQKSr8lkBmUUPiA=="}' 2024-09-18T11:58:26.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 781c6a39-5570-465b-9fc0-fa8d3639debb -i td/ceph-helpers/1/new.json 2024-09-18T11:58:26.550 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:58:26.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-18T11:58:26.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBiwOpmoGidBhAAnIBcklqKQKSr8lkBmUUPiA== --osd-uuid 781c6a39-5570-465b-9fc0-fa8d3639debb 2024-09-18T11:58:26.585 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:58:26.584+0000 7fdfe3813440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:58:26.617 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:58:26.616+0000 7fdfe3813440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:58:26.618 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:58:26.617+0000 7fdfe3813440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:58:26.618 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:58:26.617+0000 7fdfe3813440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:58:26.618 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:58:26.617+0000 7fdfe3813440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:58:26.619 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:58:26.617+0000 7fdfe3813440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:58:26.657 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:58:26.656+0000 7fdfe3813440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-18T11:58:32.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-18T11:58:32.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:58:32.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:58:32.557 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T11:58:32.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:58:33.045 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T11:58:33.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:58:33.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:58:33.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:58:33.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:58:33.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:58:33.067 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:58:33.065+0000 7f8783935440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:58:33.069 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:58:33.067+0000 7f8783935440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:58:33.070 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:58:33.068+0000 7f8783935440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:58:33.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:58:33.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:58:33.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:58:33.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:58:33.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:58:33.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:33.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:58:33.397 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:58:33.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:33.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:33.582 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:58:33.581+0000 7f8783935440 -1 Falling back to public interface 2024-09-18T11:58:33.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:34.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:34.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:34.785 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:58:34.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:58:34.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:34.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:35.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:36.073 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:58:36.071+0000 7f8783935440 -1 osd.1 0 log_to_monitors true 2024-09-18T11:58:36.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:36.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:36.148 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:58:36.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:58:36.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:36.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:36.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:37.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:37.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:37.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:58:37.793 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:58:37.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:37.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:38.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:39.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:39.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:39.179 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:58:39.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:58:39.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:39.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:39.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:40.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:40.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:40.595 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:58:40.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:58:40.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:40.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:41.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:42.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:42.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:42.069 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:58:42.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:58:42.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:42.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:42.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:43.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:43.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:43.459 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:58:43.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:58:43.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:43.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:43.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:44.865 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:44.865 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:44.865 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:58:44.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:58:44.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:44.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:45.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:46.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:46.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:46.262 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:58:46.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:58:46.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:46.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:46.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:47.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:47.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:47.624 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:58:47.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:58:47.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:47.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:48.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:49.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:49.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:49.009 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:58:49.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:58:49.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:49.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:49.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:50.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:50.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:50.762 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:58:50.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:58:50.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:50.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:51.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:52.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:52.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:52.933 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:58:52.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:58:52.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:52.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:53.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:54.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:54.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:54.807 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:58:54.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:58:54.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:54.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:55.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:56.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:56.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:56.683 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:58:56.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:58:56.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:56.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:57.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:58:58.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:58:58.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:58:58.439 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:58:58.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:58:58.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:58:58.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:58:59.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:00.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:00.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:00.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:59:00.126 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:59:00.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:00.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:59:00.548 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/162740,v1:127.0.0.1:6811/162740] [v2:127.0.0.1:6812/162740,v1:127.0.0.1:6813/162740] exists,up 781c6a39-5570-465b-9fc0-fa8d3639debb 2024-09-18T11:59:00.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:59:00.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:59:00.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:59:00.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1001: test_wait_for_osd: wait_for_osd up 0 2024-09-18T11:59:00.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:59:00.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:59:00.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:59:00.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:59:00.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:00.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:59:00.550 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:59:00.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:00.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:00.932 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/160985,v1:127.0.0.1:6803/160985] [v2:127.0.0.1:6804/160985,v1:127.0.0.1:6805/160985] exists,up 378ba3a6-1d3d-47e1-a984-73f88220d462 2024-09-18T11:59:00.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:59:00.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:59:00.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:59:00.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1002: test_wait_for_osd: wait_for_osd up 1 2024-09-18T11:59:00.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:59:00.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:59:00.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:59:00.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:59:00.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:00.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:59:00.935 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:59:00.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:00.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:59:01.309 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/162740,v1:127.0.0.1:6811/162740] [v2:127.0.0.1:6812/162740,v1:127.0.0.1:6813/162740] exists,up 781c6a39-5570-465b-9fc0-fa8d3639debb 2024-09-18T11:59:01.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:59:01.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:59:01.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:59:01.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1003: test_wait_for_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-09-18T11:59:01.310 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:59:01.310 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:59:01.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:59:01.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:59:01.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:59:01.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:59:01.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1004: test_wait_for_osd: wait_for_osd down 0 2024-09-18T11:59:01.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-09-18T11:59:01.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:59:01.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:59:01.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:59:01.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:01.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:59:01.618 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:59:01.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:01.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-09-18T11:59:01.967 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 12 last_clean_interval [0,0) [v2:127.0.0.1:6802/160985,v1:127.0.0.1:6803/160985] [v2:127.0.0.1:6804/160985,v1:127.0.0.1:6805/160985] exists 378ba3a6-1d3d-47e1-a984-73f88220d462 2024-09-18T11:59:01.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:59:01.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:59:01.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:59:01.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1005: test_wait_for_osd: TIMEOUT=1 2024-09-18T11:59:01.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1005: test_wait_for_osd: wait_for_osd up 0 2024-09-18T11:59:01.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:59:01.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:59:01.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:59:01.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:59:01.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 1 )) 2024-09-18T11:59:01.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:59:01.969 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:59:01.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:01.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:02.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:03.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:03.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 1 )) 2024-09-18T11:59:03.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 1 2024-09-18T11:59:03.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1006: test_wait_for_osd: teardown td/ceph-helpers 2024-09-18T11:59:03.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:59:03.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:59:03.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:59:03.361 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:59:03.361 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:59:03.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:59:03.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:59:03.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:59:03.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:59:03.469 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:59:03.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:59:03.470 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:59:03.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:59:03.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:59:03.472 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:59:03.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:59:03.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:59:03.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:59:03.473 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:59:03.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:59:03.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:59:03.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:59:03.486 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:59:03.486 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:59:03.486 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:59:03.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:59:03.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:59:03.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:59:03.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T11:59:03.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_wait_for_peered td/ceph-helpers 2024-09-18T11:59:03.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: test_wait_for_peered: local dir=td/ceph-helpers 2024-09-18T11:59:03.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1736: test_wait_for_peered: setup td/ceph-helpers 2024-09-18T11:59:03.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T11:59:03.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T11:59:03.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T11:59:03.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:59:03.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T11:59:03.489 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:59:03.489 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:59:03.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:59:03.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:59:03.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:59:03.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:59:03.490 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:59:03.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:59:03.491 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:59:03.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:59:03.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:59:03.493 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:59:03.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:59:03.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:59:03.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:59:03.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:59:03.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:59:03.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:59:03.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T11:59:03.497 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:59:03.497 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:59:03.497 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:59:03.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T11:59:03.498 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:59:03.498 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:59:03.498 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T11:59:03.500 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:59:03.500 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:59:03.500 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:59:03.500 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T11:59:03.502 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:59:03.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:59:03.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:59:03.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T11:59:03.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1737: test_wait_for_peered: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-18T11:59:03.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T11:59:03.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:59:03.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:59:03.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:59:03.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T11:59:03.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-18T11:59:03.877 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:59:03.877 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:59:03.877 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:59:03.877 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:59:03.878 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:59:03.878 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:59:03.878 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:59:03.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-18T11:59:04.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:59:04.137 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:59:04.137 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:59:04.137 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:59:04.137 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:59:04.138 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:59:04.138 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:59:04.138 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:59:04.138 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:59:04.139 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:59:04.139 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:59:04.139 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:59:04.139 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:59:04.139 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:59:04.139 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T11:59:04.221 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:59:04.221 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:59:04.221 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:59:04.221 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:59:04.222 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:59:04.222 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:59:04.222 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:59:04.222 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:59:04.222 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:59:04.222 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:59:04.222 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:59:04.222 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T11:59:04.222 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:59:04.223 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T11:59:04.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1738: test_wait_for_peered: run_osd td/ceph-helpers 0 2024-09-18T11:59:04.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:59:04.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:59:04.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:59:04.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:59:04.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T11:59:04.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:59:04.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:59:04.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:59:04.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:59:04.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T11:59:04.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T11:59:04.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:59:04.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:59:04.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:59:04.300 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:59:04.300 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:59:04.300 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:59:04.300 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:59:04.300 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:59:04.300 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:59:04.300 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:59:04.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:59:04.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:59:04.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:59:04.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:59:04.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:59:04.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:59:04.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:59:04.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:59:04.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:59:04.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:59:04.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:59:04.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:59:04.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T11:59:04.301 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:59:04.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=073d449b-cd10-4456-925a-82f47ef089e3 2024-09-18T11:59:04.302 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 073d449b-cd10-4456-925a-82f47ef089e3 2024-09-18T11:59:04.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 073d449b-cd10-4456-925a-82f47ef089e3' 2024-09-18T11:59:04.303 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:59:04.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCIwOpmOE+lEhAALzArbJCESNRXxEzEIKtwRQ== 2024-09-18T11:59:04.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCIwOpmOE+lEhAALzArbJCESNRXxEzEIKtwRQ=="}' 2024-09-18T11:59:04.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 073d449b-cd10-4456-925a-82f47ef089e3 -i td/ceph-helpers/0/new.json 2024-09-18T11:59:05.179 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:59:05.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T11:59:05.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCIwOpmOE+lEhAALzArbJCESNRXxEzEIKtwRQ== --osd-uuid 073d449b-cd10-4456-925a-82f47ef089e3 2024-09-18T11:59:05.214 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:05.212+0000 7f1d9b521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:59:05.227 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:05.226+0000 7f1d9b521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:59:05.228 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:05.227+0000 7f1d9b521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:59:05.229 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:05.227+0000 7f1d9b521440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:59:05.229 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:05.227+0000 7f1d9b521440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:59:05.229 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:05.227+0000 7f1d9b521440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T11:59:05.268 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:05.267+0000 7f1d9b521440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T11:59:10.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T11:59:10.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:59:10.870 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T11:59:10.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:59:10.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:59:11.066 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T11:59:11.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:59:11.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:59:11.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:59:11.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:59:11.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:59:11.089 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:11.087+0000 7f1d77523440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:59:11.094 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:11.093+0000 7f1d77523440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:59:11.095 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:11.094+0000 7f1d77523440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:59:11.265 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:59:11.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:59:11.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:59:11.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:59:11.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:59:11.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:11.266 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:59:11.267 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:59:11.267 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:11.267 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:11.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:11.626 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:11.625+0000 7f1d77523440 -1 Falling back to public interface 2024-09-18T11:59:12.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:12.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:12.475 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:59:12.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:59:12.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:12.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:12.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:13.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:13.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:13.659 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:59:13.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:59:13.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:13.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:13.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:14.228 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:14.226+0000 7f1d77523440 -1 osd.0 0 log_to_monitors true 2024-09-18T11:59:14.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:14.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:14.845 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T11:59:14.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:59:14.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:14.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:15.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:16.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:16.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:16.224 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T11:59:16.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:59:16.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:16.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:16.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:17.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:17.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:17.413 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T11:59:17.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:59:17.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:17.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:17.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:18.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:18.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:18.598 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T11:59:18.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:59:18.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:18.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:18.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:19.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:19.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:19.785 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T11:59:19.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:59:19.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:19.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:19.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:20.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:20.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:20.976 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T11:59:20.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:59:20.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:20.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:21.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:22.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:22.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:22.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:59:22.165 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T11:59:22.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:22.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:22.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:23.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:23.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:23.352 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T11:59:23.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:59:23.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:23.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:23.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:24.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:24.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:24.541 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T11:59:24.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:59:24.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:24.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:24.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:25.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:25.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:25.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:59:25.732 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T11:59:25.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:25.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:25.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:26.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:26.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:26.919 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T11:59:26.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:59:26.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:26.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:27.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:28.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:28.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:28.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:59:28.108 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T11:59:28.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:28.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:28.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:29.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:29.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:29.299 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T11:59:29.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:59:29.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:29.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:29.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:30.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:30.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:30.486 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T11:59:30.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:59:30.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:30.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:30.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:31.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:31.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:31.676 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T11:59:31.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:59:31.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:31.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:31.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:32.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:32.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:32.864 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T11:59:32.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:59:32.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:32.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:33.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:34.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:34.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:34.055 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-18T11:59:34.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T11:59:34.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:34.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:34.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:35.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:35.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:35.240 INFO:tasks.workunit.client.0.smithi157.stdout:20 2024-09-18T11:59:35.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T11:59:35.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:35.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:59:35.510 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/164727,v1:127.0.0.1:6801/164727] [v2:127.0.0.1:6802/164727,v1:127.0.0.1:6803/164727] exists,up 073d449b-cd10-4456-925a-82f47ef089e3 2024-09-18T11:59:35.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:59:35.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:59:35.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:59:35.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1739: test_wait_for_peered: run_mgr td/ceph-helpers x 2024-09-18T11:59:35.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T11:59:35.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:59:35.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:59:35.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:59:35.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T11:59:35.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:59:35.806 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:59:35.806 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:59:35.806 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:59:35.806 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:59:35.807 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:59:35.807 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:59:35.807 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:59:35.807 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:59:35.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:59:35.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1740: test_wait_for_peered: create_rbd_pool 2024-09-18T11:59:35.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T11:59:36.151 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T11:59:36.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T11:59:36.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:59:36.528 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T11:59:36.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:59:37.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T11:59:38.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1741: test_wait_for_peered: WAIT_FOR_CLEAN_TIMEOUT=1 2024-09-18T11:59:38.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1741: test_wait_for_peered: wait_for_clean 2024-09-18T11:59:38.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:59:38.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:59:38.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:59:38.653 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 1 .1 2024-09-18T11:59:38.653 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:59:38.653 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:59:38.653 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:59:38.653 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:59:38.654 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:59:38.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2024-09-18T11:59:38.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:59:38.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:59:38.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:59:38.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:59:38.679 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:59:38.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T11:59:38.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:59:38.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:59:38.862 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:59:38.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=0 2024-09-18T11:59:38.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 0 2024-09-18T11:59:38.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-0' 2024-09-18T11:59:38.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:59:38.999 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-0 2024-09-18T11:59:38.999 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:59:39.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:59:39.000 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-0 2024-09-18T11:59:39.001 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:59:39.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=0 2024-09-18T11:59:39.002 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 0 2024-09-18T11:59:39.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 0' 2024-09-18T11:59:39.002 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:59:39.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 0 -lt 0 2024-09-18T11:59:39.183 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:59:39.184 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:59:39.184 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:59:39.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-09-18T11:59:39.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-09-18T11:59:40.470 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:59:40.470 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:59:40.470 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:59:40.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T11:59:40.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:59:40.953 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:59:40.953 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:59:40.953 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:59:40.954 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:59:40.954 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:59:40.954 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:59:41.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T11:59:41.298 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:59:41.299 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:59:41.299 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:59:41.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-18T11:59:41.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T11:59:41.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T11:59:41.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T11:59:41.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:59:41.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T11:59:41.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:59:41.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:59:41.872 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:59:41.872 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:59:41.872 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:59:41.872 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:59:41.872 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:59:41.872 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:59:42.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T11:59:42.216 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:59:42.216 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:59:42.216 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:59:42.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-18T11:59:42.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T11:59:42.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T11:59:42.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T11:59:42.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T11:59:42.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T11:59:42.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T11:59:42.690 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T11:59:42.691 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T11:59:43.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T11:59:43.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T11:59:43.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 4 )) 2024-09-18T11:59:43.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:59:43.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T11:59:43.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:59:43.359 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:59:43.359 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:59:43.359 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:59:43.359 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:59:43.359 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:59:43.359 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:59:43.359 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:59:43.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T11:59:43.705 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:59:43.705 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:59:43.705 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:59:44.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-18T11:59:44.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T11:59:44.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T11:59:44.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T11:59:44.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T11:59:44.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T11:59:44.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T11:59:44.171 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T11:59:44.171 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T11:59:44.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T11:59:44.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T11:59:44.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 4 )) 2024-09-18T11:59:44.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:59:44.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-18T11:59:45.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:59:45.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:59:45.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:59:45.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:59:45.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:59:45.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:59:45.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:59:45.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:59:45.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T11:59:45.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:59:45.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:59:45.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:59:45.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-18T11:59:45.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T11:59:45.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T11:59:45.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T11:59:45.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T11:59:45.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T11:59:45.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T11:59:45.860 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T11:59:45.860 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T11:59:46.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T11:59:46.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T11:59:46.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 4 )) 2024-09-18T11:59:46.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:59:46.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.3 2024-09-18T11:59:46.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:59:46.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:59:46.636 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:59:46.636 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:59:46.636 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:59:46.636 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:59:46.636 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:59:46.636 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:59:46.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T11:59:46.976 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:59:46.976 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:59:46.977 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:59:47.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-18T11:59:47.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T11:59:47.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T11:59:47.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T11:59:47.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T11:59:47.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T11:59:47.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T11:59:47.477 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T11:59:47.477 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T11:59:47.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T11:59:47.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T11:59:47.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 4 )) 2024-09-18T11:59:47.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-18T11:59:48.436 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-18T11:59:48.436 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_fingerprint": "04aef9df-7f3b-461e-9fdf-ba7e86a00bc6", 2024-09-18T11:59:48.436 INFO:tasks.workunit.client.0.smithi157.stdout: "version": "17.2.7-1645-g7061ea6c", 2024-09-18T11:59:48.436 INFO:tasks.workunit.client.0.smithi157.stdout: "commit": "7061ea6ca50908d104e9f3979dc615e118d9e153", 2024-09-18T11:59:48.436 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-18T11:59:48.434304+0000", 2024-09-18T11:59:48.436 INFO:tasks.workunit.client.0.smithi157.stdout: "tag": "", 2024-09-18T11:59:48.436 INFO:tasks.workunit.client.0.smithi157.stdout: "health": { 2024-09-18T11:59:48.436 INFO:tasks.workunit.client.0.smithi157.stdout: "status": "HEALTH_WARN", 2024-09-18T11:59:48.436 INFO:tasks.workunit.client.0.smithi157.stdout: "checks": { 2024-09-18T11:59:48.437 INFO:tasks.workunit.client.0.smithi157.stdout: "PG_DEGRADED": { 2024-09-18T11:59:48.437 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-18T11:59:48.437 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-18T11:59:48.437 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded", 2024-09-18T11:59:48.437 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 2 2024-09-18T11:59:48.437 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.437 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-18T11:59:48.437 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.437 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2024-09-18T11:59:48.437 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.437 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.437 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0]" 2024-09-18T11:59:48.438 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.438 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.438 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-18T11:59:48.438 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.438 INFO:tasks.workunit.client.0.smithi157.stdout: "TOO_FEW_OSDS": { 2024-09-18T11:59:48.438 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-18T11:59:48.438 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-18T11:59:48.438 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2024-09-18T11:59:48.438 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 2 2024-09-18T11:59:48.438 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.438 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [], 2024-09-18T11:59:48.438 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-18T11:59:48.438 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.439 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.439 INFO:tasks.workunit.client.0.smithi157.stdout: "mutes": [] 2024-09-18T11:59:48.439 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.439 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_first_committed": 1, 2024-09-18T11:59:48.439 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_last_committed": 1, 2024-09-18T11:59:48.439 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap": { 2024-09-18T11:59:48.439 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-18T11:59:48.439 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "92255036-ee37-4797-aa7f-f8e6b3f0325b", 2024-09-18T11:59:48.439 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-18T11:59:03.520727Z", 2024-09-18T11:59:48.439 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-18T11:59:03.520727Z", 2024-09-18T11:59:48.439 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release": 17, 2024-09-18T11:59:48.439 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:59:48.439 INFO:tasks.workunit.client.0.smithi157.stdout: "election_strategy": 1, 2024-09-18T11:59:48.440 INFO:tasks.workunit.client.0.smithi157.stdout: "disallowed_leaders: ": "", 2024-09-18T11:59:48.440 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": false, 2024-09-18T11:59:48.440 INFO:tasks.workunit.client.0.smithi157.stdout: "tiebreaker_mon": "", 2024-09-18T11:59:48.440 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_ranks: ": "", 2024-09-18T11:59:48.440 INFO:tasks.workunit.client.0.smithi157.stdout: "features": { 2024-09-18T11:59:48.440 INFO:tasks.workunit.client.0.smithi157.stdout: "persistent": [ 2024-09-18T11:59:48.440 INFO:tasks.workunit.client.0.smithi157.stdout: "kraken", 2024-09-18T11:59:48.440 INFO:tasks.workunit.client.0.smithi157.stdout: "luminous", 2024-09-18T11:59:48.440 INFO:tasks.workunit.client.0.smithi157.stdout: "mimic", 2024-09-18T11:59:48.440 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap-prune", 2024-09-18T11:59:48.440 INFO:tasks.workunit.client.0.smithi157.stdout: "nautilus", 2024-09-18T11:59:48.440 INFO:tasks.workunit.client.0.smithi157.stdout: "octopus", 2024-09-18T11:59:48.441 INFO:tasks.workunit.client.0.smithi157.stdout: "pacific", 2024-09-18T11:59:48.441 INFO:tasks.workunit.client.0.smithi157.stdout: "elector-pinging", 2024-09-18T11:59:48.441 INFO:tasks.workunit.client.0.smithi157.stdout: "quincy" 2024-09-18T11:59:48.441 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.441 INFO:tasks.workunit.client.0.smithi157.stdout: "optional": [] 2024-09-18T11:59:48.441 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.441 INFO:tasks.workunit.client.0.smithi157.stdout: "mons": [ 2024-09-18T11:59:48.441 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.441 INFO:tasks.workunit.client.0.smithi157.stdout: "rank": 0, 2024-09-18T11:59:48.441 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "a", 2024-09-18T11:59:48.442 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-18T11:59:48.442 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-18T11:59:48.442 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.442 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-18T11:59:48.442 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7109", 2024-09-18T11:59:48.442 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 0 2024-09-18T11:59:48.442 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.442 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:59:48.442 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.442 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7109/0", 2024-09-18T11:59:48.442 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:7109/0", 2024-09-18T11:59:48.443 INFO:tasks.workunit.client.0.smithi157.stdout: "priority": 0, 2024-09-18T11:59:48.443 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 0, 2024-09-18T11:59:48.443 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_location": "{}" 2024-09-18T11:59:48.443 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.443 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:59:48.443 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.443 INFO:tasks.workunit.client.0.smithi157.stdout: "quorum": [ 2024-09-18T11:59:48.443 INFO:tasks.workunit.client.0.smithi157.stdout: 0 2024-09-18T11:59:48.443 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.443 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap": { 2024-09-18T11:59:48.443 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 13, 2024-09-18T11:59:48.443 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "92255036-ee37-4797-aa7f-f8e6b3f0325b", 2024-09-18T11:59:48.444 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-18T11:59:04.389727+0000", 2024-09-18T11:59:48.444 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-18T11:59:38.270603+0000", 2024-09-18T11:59:48.444 INFO:tasks.workunit.client.0.smithi157.stdout: "last_up_change": "2024-09-18T11:59:35.273772+0000", 2024-09-18T11:59:48.444 INFO:tasks.workunit.client.0.smithi157.stdout: "last_in_change": "2024-09-18T11:59:05.061452+0000", 2024-09-18T11:59:48.444 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T11:59:48.444 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_num": 5799936, 2024-09-18T11:59:48.444 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_set": [ 2024-09-18T11:59:48.444 INFO:tasks.workunit.client.0.smithi157.stdout: "pglog_hardlimit", 2024-09-18T11:59:48.444 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snapdirs", 2024-09-18T11:59:48.444 INFO:tasks.workunit.client.0.smithi157.stdout: "recovery_deletes", 2024-09-18T11:59:48.444 INFO:tasks.workunit.client.0.smithi157.stdout: "sortbitwise" 2024-09-18T11:59:48.444 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.444 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_version": 3, 2024-09-18T11:59:48.445 INFO:tasks.workunit.client.0.smithi157.stdout: "full_ratio": 0.99000000953674316, 2024-09-18T11:59:48.445 INFO:tasks.workunit.client.0.smithi157.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-18T11:59:48.445 INFO:tasks.workunit.client.0.smithi157.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T11:59:48.445 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_snapshot": "", 2024-09-18T11:59:48.445 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_max": 1, 2024-09-18T11:59:48.445 INFO:tasks.workunit.client.0.smithi157.stdout: "max_osd": 1, 2024-09-18T11:59:48.445 INFO:tasks.workunit.client.0.smithi157.stdout: "require_min_compat_client": "luminous", 2024-09-18T11:59:48.445 INFO:tasks.workunit.client.0.smithi157.stdout: "min_compat_client": "jewel", 2024-09-18T11:59:48.445 INFO:tasks.workunit.client.0.smithi157.stdout: "require_osd_release": "quincy", 2024-09-18T11:59:48.445 INFO:tasks.workunit.client.0.smithi157.stdout: "pools": [ 2024-09-18T11:59:48.445 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.445 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-18T11:59:48.445 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_name": "rbd", 2024-09-18T11:59:48.446 INFO:tasks.workunit.client.0.smithi157.stdout: "create_time": "2024-09-18T11:59:36.321623+0000", 2024-09-18T11:59:48.446 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": 8193, 2024-09-18T11:59:48.446 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T11:59:48.446 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 1, 2024-09-18T11:59:48.446 INFO:tasks.workunit.client.0.smithi157.stdout: "size": 2, 2024-09-18T11:59:48.446 INFO:tasks.workunit.client.0.smithi157.stdout: "min_size": 1, 2024-09-18T11:59:48.446 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_rule": 0, 2024-09-18T11:59:48.446 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_count": 0, 2024-09-18T11:59:48.446 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_target": 0, 2024-09-18T11:59:48.447 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T11:59:48.447 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T11:59:48.447 INFO:tasks.workunit.client.0.smithi157.stdout: "object_hash": 2, 2024-09-18T11:59:48.447 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_autoscale_mode": "off", 2024-09-18T11:59:48.447 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num": 4, 2024-09-18T11:59:48.447 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num": 4, 2024-09-18T11:59:48.447 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num_target": 4, 2024-09-18T11:59:48.447 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_target": 4, 2024-09-18T11:59:48.447 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_pending": 4, 2024-09-18T11:59:48.447 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pg_merge_meta": { 2024-09-18T11:59:48.447 INFO:tasks.workunit.client.0.smithi157.stdout: "source_pgid": "0.0", 2024-09-18T11:59:48.447 INFO:tasks.workunit.client.0.smithi157.stdout: "ready_epoch": 0, 2024-09-18T11:59:48.448 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_started": 0, 2024-09-18T11:59:48.448 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": 0, 2024-09-18T11:59:48.448 INFO:tasks.workunit.client.0.smithi157.stdout: "source_version": "0'0", 2024-09-18T11:59:48.448 INFO:tasks.workunit.client.0.smithi157.stdout: "target_version": "0'0" 2024-09-18T11:59:48.448 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.448 INFO:tasks.workunit.client.0.smithi157.stdout: "last_change": "13", 2024-09-18T11:59:48.449 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend": "0", 2024-09-18T11:59:48.449 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T11:59:48.449 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T11:59:48.449 INFO:tasks.workunit.client.0.smithi157.stdout: "auid": 0, 2024-09-18T11:59:48.449 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_mode": "selfmanaged", 2024-09-18T11:59:48.449 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_seq": 3, 2024-09-18T11:59:48.450 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_epoch": 13, 2024-09-18T11:59:48.450 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_snaps": [], 2024-09-18T11:59:48.450 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps": "[]", 2024-09-18T11:59:48.450 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_bytes": 0, 2024-09-18T11:59:48.450 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_objects": 0, 2024-09-18T11:59:48.450 INFO:tasks.workunit.client.0.smithi157.stdout: "tiers": [], 2024-09-18T11:59:48.450 INFO:tasks.workunit.client.0.smithi157.stdout: "tier_of": -1, 2024-09-18T11:59:48.450 INFO:tasks.workunit.client.0.smithi157.stdout: "read_tier": -1, 2024-09-18T11:59:48.450 INFO:tasks.workunit.client.0.smithi157.stdout: "write_tier": -1, 2024-09-18T11:59:48.450 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_mode": "none", 2024-09-18T11:59:48.450 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_bytes": 0, 2024-09-18T11:59:48.450 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_objects": 0, 2024-09-18T11:59:48.451 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T11:59:48.451 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T11:59:48.451 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T11:59:48.451 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_flush_age": 0, 2024-09-18T11:59:48.451 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_evict_age": 0, 2024-09-18T11:59:48.452 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profile": "", 2024-09-18T11:59:48.452 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_params": { 2024-09-18T11:59:48.453 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "none" 2024-09-18T11:59:48.453 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.456 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_period": 0, 2024-09-18T11:59:48.457 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_count": 0, 2024-09-18T11:59:48.457 INFO:tasks.workunit.client.0.smithi157.stdout: "use_gmt_hitset": true, 2024-09-18T11:59:48.457 INFO:tasks.workunit.client.0.smithi157.stdout: "min_read_recency_for_promote": 0, 2024-09-18T11:59:48.457 INFO:tasks.workunit.client.0.smithi157.stdout: "min_write_recency_for_promote": 0, 2024-09-18T11:59:48.457 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T11:59:48.457 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_search_last_n": 0, 2024-09-18T11:59:48.457 INFO:tasks.workunit.client.0.smithi157.stdout: "grade_table": [], 2024-09-18T11:59:48.457 INFO:tasks.workunit.client.0.smithi157.stdout: "stripe_width": 0, 2024-09-18T11:59:48.457 INFO:tasks.workunit.client.0.smithi157.stdout: "expected_num_objects": 0, 2024-09-18T11:59:48.457 INFO:tasks.workunit.client.0.smithi157.stdout: "fast_read": false, 2024-09-18T11:59:48.457 INFO:tasks.workunit.client.0.smithi157.stdout: "options": {}, 2024-09-18T11:59:48.457 INFO:tasks.workunit.client.0.smithi157.stdout: "application_metadata": { 2024-09-18T11:59:48.457 INFO:tasks.workunit.client.0.smithi157.stdout: "rbd": {} 2024-09-18T11:59:48.457 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.458 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.458 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.458 INFO:tasks.workunit.client.0.smithi157.stdout: "osds": [ 2024-09-18T11:59:48.458 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.458 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-18T11:59:48.458 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "073d449b-cd10-4456-925a-82f47ef089e3", 2024-09-18T11:59:48.458 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 1, 2024-09-18T11:59:48.459 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-18T11:59:48.459 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-18T11:59:48.459 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-18T11:59:48.459 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 0, 2024-09-18T11:59:48.459 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 0, 2024-09-18T11:59:48.459 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 5, 2024-09-18T11:59:48.459 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 7, 2024-09-18T11:59:48.459 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 0, 2024-09-18T11:59:48.459 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-18T11:59:48.459 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-18T11:59:48.460 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-18T11:59:48.460 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.460 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-18T11:59:48.460 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6800", 2024-09-18T11:59:48.460 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 164727 2024-09-18T11:59:48.460 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.460 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.460 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-18T11:59:48.460 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6801", 2024-09-18T11:59:48.460 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 164727 2024-09-18T11:59:48.460 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.460 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:59:48.460 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.460 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-18T11:59:48.461 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-18T11:59:48.461 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.461 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-18T11:59:48.461 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6802", 2024-09-18T11:59:48.461 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 164727 2024-09-18T11:59:48.461 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.461 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.461 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-18T11:59:48.461 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6803", 2024-09-18T11:59:48.461 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 164727 2024-09-18T11:59:48.461 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.461 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:59:48.461 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.461 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-18T11:59:48.462 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-18T11:59:48.462 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.462 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-18T11:59:48.462 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6806", 2024-09-18T11:59:48.462 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 164727 2024-09-18T11:59:48.462 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.462 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.462 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-18T11:59:48.462 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6807", 2024-09-18T11:59:48.462 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 164727 2024-09-18T11:59:48.462 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.462 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:59:48.462 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.463 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-18T11:59:48.463 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-18T11:59:48.463 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.463 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-18T11:59:48.463 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6804", 2024-09-18T11:59:48.463 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 164727 2024-09-18T11:59:48.463 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.463 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.463 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-18T11:59:48.463 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6805", 2024-09-18T11:59:48.463 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 164727 2024-09-18T11:59:48.463 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.463 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:59:48.463 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.464 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6801/164727", 2024-09-18T11:59:48.464 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6803/164727", 2024-09-18T11:59:48.464 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6807/164727", 2024-09-18T11:59:48.464 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6805/164727", 2024-09-18T11:59:48.464 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-18T11:59:48.464 INFO:tasks.workunit.client.0.smithi157.stdout: "exists", 2024-09-18T11:59:48.464 INFO:tasks.workunit.client.0.smithi157.stdout: "up" 2024-09-18T11:59:48.464 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:59:48.464 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.464 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.464 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_xinfo": [ 2024-09-18T11:59:48.464 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.464 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-18T11:59:48.464 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "0.000000", 2024-09-18T11:59:48.465 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-18T11:59:48.465 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-18T11:59:48.465 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540138320759226367, 2024-09-18T11:59:48.465 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-18T11:59:48.465 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-18T11:59:34.988196+0000", 2024-09-18T11:59:48.465 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-18T11:59:48.465 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.465 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.465 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap": [], 2024-09-18T11:59:48.465 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap_items": [], 2024-09-18T11:59:48.465 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_temp": [], 2024-09-18T11:59:48.465 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_temp": [], 2024-09-18T11:59:48.465 INFO:tasks.workunit.client.0.smithi157.stdout: "blocklist": {}, 2024-09-18T11:59:48.466 INFO:tasks.workunit.client.0.smithi157.stdout: "range_blocklist": {}, 2024-09-18T11:59:48.466 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profiles": { 2024-09-18T11:59:48.466 INFO:tasks.workunit.client.0.smithi157.stdout: "default": { 2024-09-18T11:59:48.466 INFO:tasks.workunit.client.0.smithi157.stdout: "k": "2", 2024-09-18T11:59:48.466 INFO:tasks.workunit.client.0.smithi157.stdout: "m": "2", 2024-09-18T11:59:48.466 INFO:tasks.workunit.client.0.smithi157.stdout: "plugin": "jerasure", 2024-09-18T11:59:48.466 INFO:tasks.workunit.client.0.smithi157.stdout: "technique": "reed_sol_van" 2024-09-18T11:59:48.466 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.466 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.466 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps_queue": [ 2024-09-18T11:59:48.466 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.466 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-18T11:59:48.466 INFO:tasks.workunit.client.0.smithi157.stdout: "snaps": [ 2024-09-18T11:59:48.467 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.467 INFO:tasks.workunit.client.0.smithi157.stdout: "begin": 2, 2024-09-18T11:59:48.467 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 1 2024-09-18T11:59:48.467 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.467 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:59:48.467 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.467 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.467 INFO:tasks.workunit.client.0.smithi157.stdout: "new_removed_snaps": [ 2024-09-18T11:59:48.467 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.467 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-18T11:59:48.467 INFO:tasks.workunit.client.0.smithi157.stdout: "snaps": [ 2024-09-18T11:59:48.467 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.467 INFO:tasks.workunit.client.0.smithi157.stdout: "begin": 2, 2024-09-18T11:59:48.467 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 1 2024-09-18T11:59:48.468 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.468 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:59:48.468 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.468 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.468 INFO:tasks.workunit.client.0.smithi157.stdout: "new_purged_snaps": [], 2024-09-18T11:59:48.468 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_node_flags": {}, 2024-09-18T11:59:48.468 INFO:tasks.workunit.client.0.smithi157.stdout: "device_class_flags": {}, 2024-09-18T11:59:48.468 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": { 2024-09-18T11:59:48.468 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_enabled": false, 2024-09-18T11:59:48.468 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_bucket_count": 0, 2024-09-18T11:59:48.468 INFO:tasks.workunit.client.0.smithi157.stdout: "degraded_stretch_mode": 0, 2024-09-18T11:59:48.468 INFO:tasks.workunit.client.0.smithi157.stdout: "recovering_stretch_mode": 0, 2024-09-18T11:59:48.468 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_bucket": 0 2024-09-18T11:59:48.468 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.469 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.469 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_metadata": [ 2024-09-18T11:59:48.469 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.469 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-18T11:59:48.469 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-18T11:59:48.469 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6802/164727,v1:127.0.0.1:6803/164727]", 2024-09-18T11:59:48.469 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-18T11:59:48.469 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-18T11:59:48.469 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-18T11:59:48.469 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T11:59:48.469 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-18T11:59:48.469 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T11:59:48.469 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T11:59:48.469 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T11:59:48.470 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T11:59:48.470 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T11:59:48.470 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T11:59:48.470 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2024-09-18T11:59:48.470 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T11:59:48.470 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T11:59:48.470 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T11:59:48.470 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T11:59:48.470 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T11:59:48.470 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "quincy", 2024-09-18T11:59:48.470 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T11:59:48.470 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T11:59:48.470 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T11:59:48.470 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-18T11:59:48.471 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-18T11:59:10.217021Z", 2024-09-18T11:59:48.471 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-18T11:59:48.471 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-18T11:59:48.471 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T11:59:48.471 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-18T11:59:48.471 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-18T11:59:48.471 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T11:59:48.471 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-18T11:59:48.471 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6800/164727,v1:127.0.0.1:6801/164727]", 2024-09-18T11:59:48.471 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-18T11:59:48.471 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/164727,v1:127.0.0.1:6807/164727]", 2024-09-18T11:59:48.471 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/164727,v1:127.0.0.1:6805/164727]", 2024-09-18T11:59:48.471 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-18T11:59:48.471 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-18T11:59:48.472 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T11:59:48.472 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T11:59:48.472 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-18T11:59:48.472 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487300", 2024-09-18T11:59:48.472 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T11:59:48.472 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T11:59:48.472 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-18T11:59:48.472 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/ceph-helpers/0", 2024-09-18T11:59:48.472 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-18T11:59:48.472 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-18T11:59:48.472 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-18T11:59:48.472 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.472 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.472 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_clean_epochs": { 2024-09-18T11:59:48.473 INFO:tasks.workunit.client.0.smithi157.stdout: "min_last_epoch_clean": 0, 2024-09-18T11:59:48.473 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": { 2024-09-18T11:59:48.473 INFO:tasks.workunit.client.0.smithi157.stdout: "per_pool": [] 2024-09-18T11:59:48.473 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.473 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_epochs": [ 2024-09-18T11:59:48.473 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.473 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-18T11:59:48.473 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 5 2024-09-18T11:59:48.473 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.473 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:59:48.473 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.473 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_first_committed": 1, 2024-09-18T11:59:48.473 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_last_committed": 13, 2024-09-18T11:59:48.473 INFO:tasks.workunit.client.0.smithi157.stdout: "crushmap": { 2024-09-18T11:59:48.474 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": [ 2024-09-18T11:59:48.474 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.474 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-18T11:59:48.474 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.0", 2024-09-18T11:59:48.474 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-18T11:59:48.474 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.474 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.474 INFO:tasks.workunit.client.0.smithi157.stdout: "types": [ 2024-09-18T11:59:48.474 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.474 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 0, 2024-09-18T11:59:48.474 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd" 2024-09-18T11:59:48.474 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.474 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.475 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-18T11:59:48.475 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "host" 2024-09-18T11:59:48.475 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.475 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.475 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 2, 2024-09-18T11:59:48.475 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "chassis" 2024-09-18T11:59:48.475 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.475 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.475 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 3, 2024-09-18T11:59:48.475 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "rack" 2024-09-18T11:59:48.475 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.476 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.476 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 4, 2024-09-18T11:59:48.476 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "row" 2024-09-18T11:59:48.476 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.476 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.476 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 5, 2024-09-18T11:59:48.476 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pdu" 2024-09-18T11:59:48.476 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.476 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.476 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 6, 2024-09-18T11:59:48.476 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pod" 2024-09-18T11:59:48.476 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.476 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.476 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 7, 2024-09-18T11:59:48.477 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "room" 2024-09-18T11:59:48.477 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.477 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.477 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 8, 2024-09-18T11:59:48.477 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "datacenter" 2024-09-18T11:59:48.477 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.477 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.477 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 9, 2024-09-18T11:59:48.477 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "zone" 2024-09-18T11:59:48.477 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.477 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.478 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 10, 2024-09-18T11:59:48.478 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "region" 2024-09-18T11:59:48.478 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.478 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.478 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-18T11:59:48.478 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "root" 2024-09-18T11:59:48.478 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.478 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.478 INFO:tasks.workunit.client.0.smithi157.stdout: "buckets": [ 2024-09-18T11:59:48.478 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.478 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -1, 2024-09-18T11:59:48.478 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default", 2024-09-18T11:59:48.478 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-18T11:59:48.478 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-18T11:59:48.479 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-18T11:59:48.479 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-18T11:59:48.479 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-18T11:59:48.479 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-18T11:59:48.479 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.479 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-18T11:59:48.479 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-18T11:59:48.479 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-18T11:59:48.479 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.479 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:59:48.479 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.479 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.479 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -2, 2024-09-18T11:59:48.479 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default~hdd", 2024-09-18T11:59:48.480 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-18T11:59:48.480 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-18T11:59:48.480 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-18T11:59:48.480 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-18T11:59:48.481 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-18T11:59:48.481 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-18T11:59:48.481 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.481 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-18T11:59:48.481 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-18T11:59:48.481 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-18T11:59:48.481 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.481 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:59:48.481 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.481 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.481 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-18T11:59:48.481 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157", 2024-09-18T11:59:48.481 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-18T11:59:48.481 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-18T11:59:48.482 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-18T11:59:48.482 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-18T11:59:48.482 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-18T11:59:48.482 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-18T11:59:48.482 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.482 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-18T11:59:48.482 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-18T11:59:48.482 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-18T11:59:48.482 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.482 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:59:48.482 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.482 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.482 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-18T11:59:48.482 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157~hdd", 2024-09-18T11:59:48.483 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-18T11:59:48.483 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-18T11:59:48.483 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-18T11:59:48.483 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-18T11:59:48.483 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-18T11:59:48.483 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-18T11:59:48.483 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.483 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-18T11:59:48.483 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-18T11:59:48.483 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-18T11:59:48.483 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.483 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:59:48.483 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.483 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.484 INFO:tasks.workunit.client.0.smithi157.stdout: "rules": [ 2024-09-18T11:59:48.484 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.484 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_id": 0, 2024-09-18T11:59:48.484 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_name": "replicated_rule", 2024-09-18T11:59:48.484 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 1, 2024-09-18T11:59:48.484 INFO:tasks.workunit.client.0.smithi157.stdout: "steps": [ 2024-09-18T11:59:48.484 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.484 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "take", 2024-09-18T11:59:48.484 INFO:tasks.workunit.client.0.smithi157.stdout: "item": -1, 2024-09-18T11:59:48.484 INFO:tasks.workunit.client.0.smithi157.stdout: "item_name": "default" 2024-09-18T11:59:48.484 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.484 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.484 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "choose_firstn", 2024-09-18T11:59:48.485 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 0, 2024-09-18T11:59:48.485 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "osd" 2024-09-18T11:59:48.485 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.485 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.485 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "emit" 2024-09-18T11:59:48.485 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.485 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-18T11:59:48.485 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.485 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.485 INFO:tasks.workunit.client.0.smithi157.stdout: "tunables": { 2024-09-18T11:59:48.485 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_tries": 0, 2024-09-18T11:59:48.485 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_fallback_tries": 0, 2024-09-18T11:59:48.485 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_total_tries": 50, 2024-09-18T11:59:48.485 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_descend_once": 1, 2024-09-18T11:59:48.485 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_vary_r": 1, 2024-09-18T11:59:48.486 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_stable": 1, 2024-09-18T11:59:48.486 INFO:tasks.workunit.client.0.smithi157.stdout: "straw_calc_version": 1, 2024-09-18T11:59:48.486 INFO:tasks.workunit.client.0.smithi157.stdout: "allowed_bucket_algs": 54, 2024-09-18T11:59:48.486 INFO:tasks.workunit.client.0.smithi157.stdout: "profile": "jewel", 2024-09-18T11:59:48.486 INFO:tasks.workunit.client.0.smithi157.stdout: "optimal_tunables": 1, 2024-09-18T11:59:48.486 INFO:tasks.workunit.client.0.smithi157.stdout: "legacy_tunables": 0, 2024-09-18T11:59:48.486 INFO:tasks.workunit.client.0.smithi157.stdout: "minimum_required_version": "jewel", 2024-09-18T11:59:48.486 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables": 1, 2024-09-18T11:59:48.486 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables2": 1, 2024-09-18T11:59:48.486 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v2_rules": 0, 2024-09-18T11:59:48.486 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables3": 1, 2024-09-18T11:59:48.486 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v3_rules": 0, 2024-09-18T11:59:48.486 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v4_buckets": 1, 2024-09-18T11:59:48.486 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables5": 1, 2024-09-18T11:59:48.487 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v5_rules": 0 2024-09-18T11:59:48.487 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.487 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_args": {} 2024-09-18T11:59:48.487 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.487 INFO:tasks.workunit.client.0.smithi157.stdout: "fsmap": { 2024-09-18T11:59:48.487 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-18T11:59:48.487 INFO:tasks.workunit.client.0.smithi157.stdout: "default_fscid": -1, 2024-09-18T11:59:48.487 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": { 2024-09-18T11:59:48.487 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": {}, 2024-09-18T11:59:48.487 INFO:tasks.workunit.client.0.smithi157.stdout: "ro_compat": {}, 2024-09-18T11:59:48.487 INFO:tasks.workunit.client.0.smithi157.stdout: "incompat": { 2024-09-18T11:59:48.487 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_1": "base v0.20", 2024-09-18T11:59:48.487 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_2": "client writeable ranges", 2024-09-18T11:59:48.487 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_3": "default file layouts on dirs", 2024-09-18T11:59:48.488 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_4": "dir inode in separate object", 2024-09-18T11:59:48.488 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_5": "mds uses versioned encoding", 2024-09-18T11:59:48.488 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-18T11:59:48.488 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_8": "no anchor table", 2024-09-18T11:59:48.488 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_9": "file layout v2", 2024-09-18T11:59:48.488 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_10": "snaprealm v2" 2024-09-18T11:59:48.488 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.488 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.488 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_flags": { 2024-09-18T11:59:48.488 INFO:tasks.workunit.client.0.smithi157.stdout: "enable_multiple": true, 2024-09-18T11:59:48.488 INFO:tasks.workunit.client.0.smithi157.stdout: "ever_enabled_multiple": true 2024-09-18T11:59:48.488 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.488 INFO:tasks.workunit.client.0.smithi157.stdout: "standbys": [], 2024-09-18T11:59:48.489 INFO:tasks.workunit.client.0.smithi157.stdout: "filesystems": [] 2024-09-18T11:59:48.489 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.489 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_first_committed": 1, 2024-09-18T11:59:48.489 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_last_committed": 1, 2024-09-18T11:59:48.489 INFO:tasks.workunit.client.0.smithi157.stdout: "auth": { 2024-09-18T11:59:48.489 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-18T11:59:48.489 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 3, 2024-09-18T11:59:48.489 INFO:tasks.workunit.client.0.smithi157.stdout: "num_secrets": 1 2024-09-18T11:59:48.489 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.489 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 4, 2024-09-18T11:59:48.489 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_active": 4, 2024-09-18T11:59:48.489 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_unknown": 0, 2024-09-18T11:59:48.489 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osd": 1, 2024-09-18T11:59:48.489 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_sum": { 2024-09-18T11:59:48.490 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-18T11:59:48.490 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 19, 2024-09-18T11:59:48.490 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 2, 2024-09-18T11:59:48.490 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 0, 2024-09-18T11:59:48.490 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 4, 2024-09-18T11:59:48.490 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T11:59:48.490 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-18T11:59:48.490 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 2, 2024-09-18T11:59:48.490 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-18T11:59:48.490 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-18T11:59:48.490 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 2, 2024-09-18T11:59:48.490 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-18T11:59:48.490 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-18T11:59:48.490 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-18T11:59:48.491 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 2, 2024-09-18T11:59:48.491 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 2, 2024-09-18T11:59:48.491 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-18T11:59:48.491 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T11:59:48.491 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-18T11:59:48.491 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-18T11:59:48.491 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-18T11:59:48.491 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-18T11:59:48.491 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-18T11:59:48.491 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T11:59:48.491 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T11:59:48.491 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-18T11:59:48.491 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-18T11:59:48.491 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-18T11:59:48.492 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-18T11:59:48.492 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-18T11:59:48.492 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-18T11:59:48.492 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-18T11:59:48.492 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-18T11:59:48.492 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-18T11:59:48.492 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-18T11:59:48.492 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-18T11:59:48.492 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-18T11:59:48.492 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-18T11:59:48.492 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-18T11:59:48.492 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-18T11:59:48.492 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-18T11:59:48.492 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.493 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-18T11:59:48.493 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-18T11:59:48.493 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-18T11:59:48.493 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-18T11:59:48.493 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 0, 2024-09-18T11:59:48.493 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 0, 2024-09-18T11:59:48.493 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-18T11:59:48.493 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-18T11:59:48.493 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-18T11:59:48.493 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-18T11:59:48.493 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-18T11:59:48.493 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.493 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 3, 2024-09-18T11:59:48.493 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 3, 2024-09-18T11:59:48.494 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 4, 2024-09-18T11:59:48.494 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 4, 2024-09-18T11:59:48.494 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 0 2024-09-18T11:59:48.494 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.494 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum": { 2024-09-18T11:59:48.494 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-18T11:59:48.494 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-18T11:59:48.494 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 4, 2024-09-18T11:59:48.494 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 1, 2024-09-18T11:59:48.494 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 1, 2024-09-18T11:59:48.494 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 1, 2024-09-18T11:59:48.494 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 104857600, 2024-09-18T11:59:48.494 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 297280, 2024-09-18T11:59:48.494 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 116, 2024-09-18T11:59:48.495 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 1, 2024-09-18T11:59:48.495 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 297150, 2024-09-18T11:59:48.495 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 104560320, 2024-09-18T11:59:48.495 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-18T11:59:48.495 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 107374182400, 2024-09-18T11:59:48.495 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 107069767680, 2024-09-18T11:59:48.495 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-18T11:59:48.495 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 118784, 2024-09-18T11:59:48.495 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 24598, 2024-09-18T11:59:48.495 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-18T11:59:48.495 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-18T11:59:48.495 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-18T11:59:48.495 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 1704, 2024-09-18T11:59:48.495 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 304281944 2024-09-18T11:59:48.496 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.496 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-18T11:59:48.496 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-18T11:59:48.496 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-18T11:59:48.496 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 0, 2024-09-18T11:59:48.496 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-18T11:59:48.496 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-18T11:59:48.496 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-18T11:59:48.496 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.496 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-18T11:59:48.496 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 0, 2024-09-18T11:59:48.496 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 0, 2024-09-18T11:59:48.496 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 0, 2024-09-18T11:59:48.496 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 0 2024-09-18T11:59:48.497 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.497 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-18T11:59:48.497 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-18T11:59:48.497 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.497 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum_by_class": { 2024-09-18T11:59:48.497 INFO:tasks.workunit.client.0.smithi157.stdout: "hdd": { 2024-09-18T11:59:48.497 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-18T11:59:48.497 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-18T11:59:48.497 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 4, 2024-09-18T11:59:48.497 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 1, 2024-09-18T11:59:48.497 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 1, 2024-09-18T11:59:48.497 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 1, 2024-09-18T11:59:48.498 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 104857600, 2024-09-18T11:59:48.498 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 297280, 2024-09-18T11:59:48.498 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 116, 2024-09-18T11:59:48.498 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 1, 2024-09-18T11:59:48.498 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 297150, 2024-09-18T11:59:48.498 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 104560320, 2024-09-18T11:59:48.498 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-18T11:59:48.498 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 107374182400, 2024-09-18T11:59:48.498 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 107069767680, 2024-09-18T11:59:48.498 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-18T11:59:48.498 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 118784, 2024-09-18T11:59:48.498 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 24598, 2024-09-18T11:59:48.498 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-18T11:59:48.498 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-18T11:59:48.498 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-18T11:59:48.499 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 1704, 2024-09-18T11:59:48.499 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 304281944 2024-09-18T11:59:48.499 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.499 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-18T11:59:48.499 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-18T11:59:48.499 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-18T11:59:48.499 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 0, 2024-09-18T11:59:48.499 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-18T11:59:48.499 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-18T11:59:48.499 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-18T11:59:48.499 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.499 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-18T11:59:48.499 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 0, 2024-09-18T11:59:48.499 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 0, 2024-09-18T11:59:48.500 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 0, 2024-09-18T11:59:48.500 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 0 2024-09-18T11:59:48.500 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.500 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-18T11:59:48.500 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-18T11:59:48.500 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.500 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.500 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_stats": [ 2024-09-18T11:59:48.500 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.500 INFO:tasks.workunit.client.0.smithi157.stdout: "poolid": 1, 2024-09-18T11:59:48.500 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 4, 2024-09-18T11:59:48.500 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-18T11:59:48.500 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 19, 2024-09-18T11:59:48.500 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 2, 2024-09-18T11:59:48.501 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 0, 2024-09-18T11:59:48.501 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 4, 2024-09-18T11:59:48.501 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T11:59:48.501 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-18T11:59:48.501 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 2, 2024-09-18T11:59:48.501 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-18T11:59:48.501 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-18T11:59:48.501 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 2, 2024-09-18T11:59:48.501 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-18T11:59:48.501 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-18T11:59:48.501 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-18T11:59:48.501 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 2, 2024-09-18T11:59:48.501 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 2, 2024-09-18T11:59:48.501 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-18T11:59:48.502 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T11:59:48.502 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-18T11:59:48.502 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-18T11:59:48.502 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-18T11:59:48.502 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-18T11:59:48.502 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-18T11:59:48.502 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T11:59:48.502 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T11:59:48.502 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-18T11:59:48.502 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-18T11:59:48.502 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-18T11:59:48.502 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-18T11:59:48.502 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-18T11:59:48.502 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-18T11:59:48.503 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-18T11:59:48.503 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-18T11:59:48.503 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-18T11:59:48.503 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-18T11:59:48.503 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-18T11:59:48.503 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-18T11:59:48.503 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-18T11:59:48.503 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-18T11:59:48.503 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-18T11:59:48.503 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-18T11:59:48.503 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.503 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-18T11:59:48.503 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-18T11:59:48.503 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-18T11:59:48.503 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-18T11:59:48.504 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 4096, 2024-09-18T11:59:48.504 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 19, 2024-09-18T11:59:48.504 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-18T11:59:48.504 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-18T11:59:48.504 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-18T11:59:48.504 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-18T11:59:48.504 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-18T11:59:48.504 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.504 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 3, 2024-09-18T11:59:48.504 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 3, 2024-09-18T11:59:48.504 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 4, 2024-09-18T11:59:48.504 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 4, 2024-09-18T11:59:48.504 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 1 2024-09-18T11:59:48.504 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.505 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.505 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_stats": [ 2024-09-18T11:59:48.505 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.505 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-18T11:59:48.505 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 21474836482 2024-09-18T11:59:48.505 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.505 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.505 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_state": [ 2024-09-18T11:59:48.505 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.505 INFO:tasks.workunit.client.0.smithi157.stdout: "state": "active+undersized+degraded", 2024-09-18T11:59:48.505 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 2 2024-09-18T11:59:48.505 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.505 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.506 INFO:tasks.workunit.client.0.smithi157.stdout: "state": "active+undersized", 2024-09-18T11:59:48.506 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 2 2024-09-18T11:59:48.506 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.506 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.506 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_osd": [ 2024-09-18T11:59:48.506 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.506 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-18T11:59:48.506 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 4, 2024-09-18T11:59:48.506 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 4, 2024-09-18T11:59:48.506 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-18T11:59:48.506 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.506 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.506 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": [ 2024-09-18T11:59:48.506 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-18T11:59:48.506 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-18T11:59:48.507 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": {} 2024-09-18T11:59:48.507 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.507 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-18T11:59:48.507 INFO:tasks.workunit.client.0.smithi157.stdout: "servicemap": { 2024-09-18T11:59:48.507 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-18T11:59:48.507 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-18T11:59:04.390283+0000", 2024-09-18T11:59:48.507 INFO:tasks.workunit.client.0.smithi157.stdout: "services": {} 2024-09-18T11:59:48.507 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-18T11:59:48.507 INFO:tasks.workunit.client.0.smithi157.stdout: "paxos": { 2024-09-18T11:59:48.507 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-18T11:59:48.507 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 102, 2024-09-18T11:59:48.507 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pn": 0, 2024-09-18T11:59:48.507 INFO:tasks.workunit.client.0.smithi157.stdout: "accepted_pn": 0 2024-09-18T11:59:48.507 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-18T11:59:48.508 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-18T11:59:48.511 INFO:tasks.workunit.client.0.smithi157.stderr:report 3150998315 2024-09-18T11:59:48.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-18T11:59:48.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1742: test_wait_for_peered: run_osd td/ceph-helpers 1 2024-09-18T11:59:48.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T11:59:48.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:59:48.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:59:48.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:59:48.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-18T11:59:48.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T11:59:48.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:59:48.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:59:48.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:59:48.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-18T11:59:48.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-18T11:59:48.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:59:48.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:59:48.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T11:59:48.512 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:59:48.512 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:59:48.512 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:59:48.512 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:59:48.512 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:59:48.512 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T11:59:48.512 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:59:48.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T11:59:48.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:59:48.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:59:48.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:59:48.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T11:59:48.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T11:59:48.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:59:48.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:59:48.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:59:48.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:59:48.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:59:48.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:59:48.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-18T11:59:48.513 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:59:48.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dcc7af0a-6e16-4f46-9249-8078c16e295f 2024-09-18T11:59:48.514 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 dcc7af0a-6e16-4f46-9249-8078c16e295f 2024-09-18T11:59:48.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 dcc7af0a-6e16-4f46-9249-8078c16e295f' 2024-09-18T11:59:48.514 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:59:48.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0wOpmP96dGxAAJqvrLCkso5BNid+5Ca1p7Q== 2024-09-18T11:59:48.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0wOpmP96dGxAAJqvrLCkso5BNid+5Ca1p7Q=="}' 2024-09-18T11:59:48.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dcc7af0a-6e16-4f46-9249-8078c16e295f -i td/ceph-helpers/1/new.json 2024-09-18T11:59:48.978 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:59:48.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-18T11:59:48.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0wOpmP96dGxAAJqvrLCkso5BNid+5Ca1p7Q== --osd-uuid dcc7af0a-6e16-4f46-9249-8078c16e295f 2024-09-18T11:59:49.014 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:49.012+0000 7f8068f1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:59:49.015 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:49.014+0000 7f8068f1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:59:49.016 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:49.015+0000 7f8068f1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:59:49.017 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:49.015+0000 7f8068f1d440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:59:49.017 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:49.015+0000 7f8068f1d440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:59:49.017 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:49.015+0000 7f8068f1d440 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-18T11:59:49.094 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:49.092+0000 7f8068f1d440 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-18T11:59:55.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-18T11:59:55.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:59:55.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:59:55.437 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T11:59:55.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:59:55.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:59:55.923 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T11:59:55.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:59:55.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:59:55.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:59:55.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:59:55.945 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:55.944+0000 7f2ad4b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:59:55.947 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:55.946+0000 7f2ad4b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:59:55.948 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:55.947+0000 7f2ad4b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:59:56.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:59:56.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:59:56.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:59:56.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:59:56.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:59:56.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:56.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:59:56.279 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T11:59:56.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:56.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:59:56.469 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:56.468+0000 7f2ad4b1f440 -1 Falling back to public interface 2024-09-18T11:59:56.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:57.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:57.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:57.631 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T11:59:57.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:59:57.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:57.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:59:57.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:59:58.961 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T11:59:58.959+0000 7f2ad4b1f440 -1 osd.1 0 log_to_monitors true 2024-09-18T11:59:58.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:59:59.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:59:59.000 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T11:59:59.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:59:59.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:59:59.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:59:59.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:00.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:00.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:00.747 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:00:00.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:00:00.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:00.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:00:01.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:02.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:02.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:02.261 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T12:00:02.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T12:00:02.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:02.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:00:02.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:03.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:03.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:03.794 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T12:00:03.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T12:00:03.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:03.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:00:04.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:05.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:05.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:05.315 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T12:00:05.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T12:00:05.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:05.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:00:05.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:06.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:06.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:06.775 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T12:00:06.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T12:00:06.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:06.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:00:07.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:08.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:08.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:08.163 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T12:00:08.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T12:00:08.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:08.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:00:08.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:09.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:09.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:09.529 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T12:00:09.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T12:00:09.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:09.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:00:09.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:10.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:10.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:10.946 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T12:00:10.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T12:00:10.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:10.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:00:11.359 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:12.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:12.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:12.361 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T12:00:12.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T12:00:12.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:12.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:00:13.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:14.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:14.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:14.160 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T12:00:14.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T12:00:14.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:14.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:00:15.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:16.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:16.390 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:16.390 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T12:00:16.390 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T12:00:16.390 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:16.390 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:00:16.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:17.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:17.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:17.948 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T12:00:17.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T12:00:17.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:17.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:00:18.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:19.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:19.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:19.626 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T12:00:19.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T12:00:19.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:19.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:00:20.706 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:21.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:21.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:21.708 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T12:00:21.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T12:00:21.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:21.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:00:22.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:23.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:23.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:23.137 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T12:00:23.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T12:00:23.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:23.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:00:23.596 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 17 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/167287,v1:127.0.0.1:6811/167287] [v2:127.0.0.1:6812/167287,v1:127.0.0.1:6813/167287] exists,up dcc7af0a-6e16-4f46-9249-8078c16e295f 2024-09-18T12:00:23.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:00:23.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:00:23.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:00:23.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1743: test_wait_for_peered: wait_for_peered 2024-09-18T12:00:23.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-09-18T12:00:23.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-09-18T12:00:23.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-09-18T12:00:23.599 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-09-18T12:00:23.599 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T12:00:23.599 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T12:00:23.599 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T12:00:23.599 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T12:00:23.599 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T12:00:23.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T12:00:23.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-09-18T12:00:23.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-09-18T12:00:23.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-09-18T12:00:23.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T12:00:23.685 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T12:00:24.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T12:00:24.109 INFO:tasks.workunit.client.0.smithi157.stderr:1' 2024-09-18T12:00:24.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T12:00:24.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:00:24.109 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T12:00:24.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836491 2024-09-18T12:00:24.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836491 2024-09-18T12:00:24.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836491' 2024-09-18T12:00:24.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:00:24.282 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T12:00:24.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=73014444039 2024-09-18T12:00:24.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 73014444039 2024-09-18T12:00:24.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836491 1-73014444039' 2024-09-18T12:00:24.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:00:24.389 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836491 2024-09-18T12:00:24.389 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:00:24.390 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T12:00:24.390 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836491 2024-09-18T12:00:24.390 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:00:24.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836491 2024-09-18T12:00:24.391 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836491 2024-09-18T12:00:24.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-09-18T12:00:24.392 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T12:00:24.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836489 -lt 21474836491 2024-09-18T12:00:24.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T12:00:25.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T12:00:25.756 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T12:00:26.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836492 -lt 21474836491 2024-09-18T12:00:26.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:00:26.256 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-73014444039 2024-09-18T12:00:26.257 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:00:26.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T12:00:26.258 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-73014444039 2024-09-18T12:00:26.258 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:00:26.259 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 73014444039 2024-09-18T12:00:26.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=73014444039 2024-09-18T12:00:26.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 73014444039' 2024-09-18T12:00:26.259 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T12:00:26.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 73014444039 -lt 73014444039 2024-09-18T12:00:26.642 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-09-18T12:00:26.642 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T12:00:26.642 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T12:00:27.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 4 == 0 2024-09-18T12:00:27.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-09-18T12:00:27.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-09-18T12:00:27.162 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-09-18T12:00:27.162 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-18T12:00:27.162 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-18T12:00:27.162 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-18T12:00:27.162 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-18T12:00:27.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=4 2024-09-18T12:00:27.505 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-09-18T12:00:27.505 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T12:00:27.505 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T12:00:27.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 4 = 4 2024-09-18T12:00:27.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-09-18T12:00:27.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-09-18T12:00:27.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1744: test_wait_for_peered: teardown td/ceph-helpers 2024-09-18T12:00:27.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T12:00:27.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:00:27.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T12:00:27.974 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:00:27.975 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:00:27.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:00:27.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:00:27.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:00:28.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:00:28.080 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:00:28.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:00:28.081 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:00:28.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:00:28.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:00:28.082 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:00:28.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:00:28.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:00:28.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:00:28.084 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:00:28.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:00:28.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:00:28.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T12:00:28.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:00:28.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:00:28.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:00:28.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T12:00:28.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:00:28.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:00:28.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T12:00:28.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_wait_for_scrub td/ceph-helpers 2024-09-18T12:00:28.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: test_wait_for_scrub: local dir=td/ceph-helpers 2024-09-18T12:00:28.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1996: test_wait_for_scrub: setup td/ceph-helpers 2024-09-18T12:00:28.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T12:00:28.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T12:00:28.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T12:00:28.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:00:28.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T12:00:28.098 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:00:28.098 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:00:28.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:00:28.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:00:28.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:00:28.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:00:28.099 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:00:28.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:00:28.099 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:00:28.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:00:28.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:00:28.101 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:00:28.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:00:28.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:00:28.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:00:28.102 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:00:28.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:00:28.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:00:28.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T12:00:28.105 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:00:28.105 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:00:28.105 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:00:28.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T12:00:28.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:00:28.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:00:28.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T12:00:28.107 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T12:00:28.107 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:00:28.107 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:00:28.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T12:00:28.109 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T12:00:28.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T12:00:28.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T12:00:28.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T12:00:28.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1997: test_wait_for_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T12:00:28.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T12:00:28.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T12:00:28.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T12:00:28.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T12:00:28.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T12:00:28.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T12:00:28.480 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T12:00:28.480 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:00:28.481 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:00:28.481 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:00:28.481 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:00:28.481 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:00:28.481 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T12:00:28.481 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T12:00:28.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T12:00:28.741 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T12:00:28.741 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:00:28.741 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T12:00:28.741 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T12:00:28.741 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T12:00:28.742 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T12:00:28.742 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T12:00:28.742 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T12:00:28.742 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:00:28.743 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:00:28.743 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:00:28.743 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T12:00:28.743 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:00:28.743 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T12:00:28.819 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T12:00:28.819 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:00:28.819 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T12:00:28.819 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T12:00:28.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T12:00:28.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T12:00:28.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T12:00:28.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T12:00:28.820 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:00:28.820 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:00:28.820 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:00:28.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T12:00:28.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:00:28.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T12:00:28.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1998: test_wait_for_scrub: run_mgr td/ceph-helpers x 2024-09-18T12:00:28.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T12:00:28.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T12:00:28.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T12:00:28.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T12:00:28.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T12:00:28.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T12:00:29.726 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T12:00:29.727 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:00:29.727 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:00:29.727 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:00:29.727 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:00:29.727 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:00:29.727 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T12:00:29.727 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T12:00:29.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T12:00:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1999: test_wait_for_scrub: run_osd td/ceph-helpers 0 2024-09-18T12:00:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T12:00:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:00:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T12:00:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:00:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T12:00:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T12:00:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:00:29.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:00:29.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:00:29.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T12:00:29.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T12:00:29.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:00:29.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T12:00:29.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T12:00:29.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:00:29.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:00:29.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:00:29.756 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:00:29.756 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:00:29.756 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:00:29.756 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T12:00:29.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T12:00:29.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:00:29.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:00:29.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:00:29.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T12:00:29.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T12:00:29.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:00:29.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:00:29.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:00:29.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:00:29.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:00:29.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T12:00:29.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T12:00:29.758 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:00:29.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=db14ef73-883b-4f82-af99-b1d4bd8dcbdc 2024-09-18T12:00:29.758 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 db14ef73-883b-4f82-af99-b1d4bd8dcbdc 2024-09-18T12:00:29.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 db14ef73-883b-4f82-af99-b1d4bd8dcbdc' 2024-09-18T12:00:29.759 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:00:29.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDdwOpmCUTRLRAARUwUkxu/kzYre/TOjfp9Zg== 2024-09-18T12:00:29.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDdwOpmCUTRLRAARUwUkxu/kzYre/TOjfp9Zg=="}' 2024-09-18T12:00:29.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new db14ef73-883b-4f82-af99-b1d4bd8dcbdc -i td/ceph-helpers/0/new.json 2024-09-18T12:00:30.079 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:00:30.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T12:00:30.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDdwOpmCUTRLRAARUwUkxu/kzYre/TOjfp9Zg== --osd-uuid db14ef73-883b-4f82-af99-b1d4bd8dcbdc 2024-09-18T12:00:30.117 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:00:30.115+0000 7fb6c1523440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:00:30.125 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:00:30.123+0000 7fb6c1523440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:00:30.126 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:00:30.124+0000 7fb6c1523440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:00:30.126 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:00:30.125+0000 7fb6c1523440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T12:00:30.127 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:00:30.125+0000 7fb6c1523440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T12:00:30.127 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:00:30.125+0000 7fb6c1523440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T12:00:30.165 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:00:30.164+0000 7fb6c1523440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T12:00:35.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T12:00:35.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:00:35.753 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T12:00:35.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T12:00:35.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:00:36.232 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T12:00:36.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T12:00:36.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:00:36.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:00:36.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:00:36.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:00:36.254 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:00:36.252+0000 7f9bf7762440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:00:36.256 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:00:36.254+0000 7f9bf7762440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:00:36.257 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:00:36.255+0000 7f9bf7762440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:00:36.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T12:00:36.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:00:36.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T12:00:36.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:00:36.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:00:36.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:36.587 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:00:36.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:00:36.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:36.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:00:36.771 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:00:36.770+0000 7f9bf7762440 -1 Falling back to public interface 2024-09-18T12:00:36.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:37.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:37.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:37.970 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:00:37.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:00:37.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:37.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:00:38.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:39.266 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:00:39.265+0000 7f9bf7762440 -1 osd.0 0 log_to_monitors true 2024-09-18T12:00:39.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:39.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:39.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:00:39.330 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:00:39.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:39.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:00:39.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:40.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:40.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:40.878 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:00:40.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:00:40.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:40.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:00:41.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:42.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:42.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:42.320 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T12:00:42.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T12:00:42.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:42.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:00:42.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:43.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:43.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:43.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T12:00:43.702 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T12:00:43.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:43.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:00:44.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:45.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:45.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:45.169 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T12:00:45.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T12:00:45.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:45.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:00:45.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:46.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:46.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:46.568 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T12:00:46.568 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T12:00:46.568 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:46.568 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:00:46.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:47.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:47.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:47.979 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T12:00:47.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T12:00:47.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:47.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:00:48.390 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:49.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:49.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:49.391 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T12:00:49.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T12:00:49.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:49.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:00:49.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:50.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:50.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:50.789 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T12:00:50.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T12:00:50.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:50.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:00:51.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:52.553 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:52.553 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:52.553 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T12:00:52.553 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T12:00:52.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:52.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:00:53.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:54.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:54.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:54.135 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T12:00:54.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T12:00:54.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:54.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:00:54.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:55.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:55.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:55.853 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T12:00:55.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T12:00:55.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:55.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:00:57.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:58.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:58.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:58.077 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T12:00:58.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T12:00:58.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:58.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:00:58.920 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:00:59.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:00:59.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:00:59.922 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T12:00:59.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T12:00:59.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:00:59.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:00.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:01.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:01.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:01.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T12:01:01.647 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T12:01:01.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:01.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:02.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:03.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:03.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:03.083 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T12:01:03.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T12:01:03.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:03.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:03.465 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/169594,v1:127.0.0.1:6803/169594] [v2:127.0.0.1:6804/169594,v1:127.0.0.1:6805/169594] exists,up db14ef73-883b-4f82-af99-b1d4bd8dcbdc 2024-09-18T12:01:03.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:01:03.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:01:03.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:01:03.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2000: test_wait_for_scrub: create_rbd_pool 2024-09-18T12:01:03.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T12:01:03.817 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T12:01:03.828 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T12:01:03.828 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T12:01:04.403 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T12:01:04.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T12:01:05.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T12:01:06.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: test_wait_for_scrub: wait_for_clean 2024-09-18T12:01:06.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T12:01:06.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T12:01:06.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T12:01:06.550 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T12:01:06.550 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T12:01:06.551 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T12:01:06.551 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T12:01:06.551 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T12:01:06.551 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T12:01:06.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T12:01:06.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T12:01:06.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T12:01:06.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T12:01:06.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T12:01:06.647 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T12:01:07.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T12:01:07.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T12:01:07.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:01:07.002 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T12:01:07.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836485 2024-09-18T12:01:07.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836485 2024-09-18T12:01:07.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485' 2024-09-18T12:01:07.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:01:07.108 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836485 2024-09-18T12:01:07.108 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:01:07.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T12:01:07.109 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836485 2024-09-18T12:01:07.109 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:01:07.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836485 2024-09-18T12:01:07.111 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836485 2024-09-18T12:01:07.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-18T12:01:07.111 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T12:01:07.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836485 2024-09-18T12:01:07.501 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T12:01:07.501 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T12:01:07.502 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T12:01:07.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T12:01:07.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T12:01:07.972 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T12:01:07.973 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T12:01:07.973 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T12:01:07.973 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T12:01:07.973 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T12:01:07.973 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T12:01:08.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T12:01:08.313 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T12:01:08.313 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T12:01:08.313 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T12:01:08.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T12:01:08.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T12:01:08.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T12:01:08.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: test_wait_for_scrub: local pgid=1.0 2024-09-18T12:01:08.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2003: test_wait_for_scrub: ceph pg repair 1.0 2024-09-18T12:01:09.122 INFO:tasks.workunit.client.0.smithi157.stderr:instructing pg 1.0 on osd.0 to repair 2024-09-18T12:01:09.133 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2004: test_wait_for_scrub: get_last_scrub_stamp 1.0 2024-09-18T12:01:09.133 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T12:01:09.134 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T12:01:09.134 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T12:01:09.134 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T12:01:09.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2004: test_wait_for_scrub: local last_scrub=2024-09-18T12:01:04.125990+0000 2024-09-18T12:01:09.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: test_wait_for_scrub: wait_for_scrub 1.0 2024-09-18T12:01:04.125990+0000 2024-09-18T12:01:09.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T12:01:09.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T12:01:04.125990+0000 2024-09-18T12:01:09.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T12:01:09.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T12:01:09.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T12:01:09.538 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T12:01:09.538 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T12:01:09.538 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T12:01:09.538 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T12:01:09.538 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T12:01:09.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T12:01:04.125990+0000 '>' 2024-09-18T12:01:04.125990+0000 2024-09-18T12:01:09.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T12:01:10.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T12:01:10.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T12:01:10.876 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T12:01:10.876 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T12:01:10.876 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T12:01:10.876 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T12:01:10.876 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T12:01:11.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T12:01:04.125990+0000 '>' 2024-09-18T12:01:04.125990+0000 2024-09-18T12:01:11.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T12:01:12.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T12:01:12.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T12:01:12.228 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T12:01:12.228 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T12:01:12.228 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T12:01:12.228 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T12:01:12.229 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T12:01:12.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T12:01:04.125990+0000 '>' 2024-09-18T12:01:04.125990+0000 2024-09-18T12:01:12.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T12:01:13.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T12:01:13.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T12:01:13.636 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T12:01:13.636 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T12:01:13.636 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T12:01:13.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T12:01:13.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T12:01:13.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T12:01:10.510517+0000 '>' 2024-09-18T12:01:04.125990+0000 2024-09-18T12:01:13.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T12:01:13.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: test_wait_for_scrub: kill_daemons td/ceph-helpers KILL osd 2024-09-18T12:01:13.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:01:13.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:01:13.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:01:13.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:01:13.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:01:14.034 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:01:14.034 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2007: test_wait_for_scrub: get_last_scrub_stamp 1.0 2024-09-18T12:01:14.034 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T12:01:14.034 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T12:01:14.034 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T12:01:14.034 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T12:01:14.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2007: test_wait_for_scrub: last_scrub=2024-09-18T12:01:10.510517+0000 2024-09-18T12:01:14.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2008: test_wait_for_scrub: TIMEOUT=1 2024-09-18T12:01:14.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2008: test_wait_for_scrub: wait_for_scrub 1.0 2024-09-18T12:01:10.510517+0000 2024-09-18T12:01:14.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T12:01:14.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T12:01:10.510517+0000 2024-09-18T12:01:14.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T12:01:14.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T12:01:14.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 1 )) 2024-09-18T12:01:14.376 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T12:01:14.376 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T12:01:14.376 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T12:01:14.377 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T12:01:14.377 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T12:01:14.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T12:01:10.510517+0000 '>' 2024-09-18T12:01:10.510517+0000 2024-09-18T12:01:14.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T12:01:15.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T12:01:15.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 1 )) 2024-09-18T12:01:15.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: wait_for_scrub: return 1 2024-09-18T12:01:15.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: test_wait_for_scrub: teardown td/ceph-helpers 2024-09-18T12:01:15.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T12:01:15.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:01:15.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T12:01:15.719 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:01:15.719 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:01:15.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:01:15.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:01:15.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:01:15.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:01:15.825 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:01:15.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:01:15.827 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:01:15.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:01:15.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:01:15.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:01:15.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:01:15.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:01:15.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:01:15.829 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:01:15.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:01:15.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:01:15.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T12:01:15.838 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:01:15.838 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:15.838 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:01:15.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T12:01:15.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:01:15.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:01:15.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: run_tests: for func in $funcs 2024-09-18T12:01:15.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: run_tests: test_wait_osd_id_used_by_pgs td/ceph-helpers 2024-09-18T12:01:15.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1501: test_wait_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-09-18T12:01:15.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1503: test_wait_osd_id_used_by_pgs: setup td/ceph-helpers 2024-09-18T12:01:15.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-18T12:01:15.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-18T12:01:15.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T12:01:15.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:01:15.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T12:01:15.841 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:01:15.841 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:01:15.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:01:15.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:01:15.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:01:15.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:01:15.842 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:01:15.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:01:15.843 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:01:15.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:01:15.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:01:15.844 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:01:15.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:01:15.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:01:15.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:01:15.846 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:01:15.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:01:15.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:01:15.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T12:01:15.849 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:01:15.849 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:15.849 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:01:15.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T12:01:15.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:01:15.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:01:15.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-18T12:01:15.851 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T12:01:15.851 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:15.851 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:01:15.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76995 2024-09-18T12:01:15.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T12:01:15.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T12:01:15.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T12:01:15.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-18T12:01:15.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1504: test_wait_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T12:01:15.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-18T12:01:15.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T12:01:15.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T12:01:15.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T12:01:15.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-18T12:01:15.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T12:01:16.232 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T12:01:16.233 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:01:16.233 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:01:16.233 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:01:16.233 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:16.233 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:01:16.233 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T12:01:16.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T12:01:16.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T12:01:16.493 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T12:01:16.493 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:01:16.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T12:01:16.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T12:01:16.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T12:01:16.494 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T12:01:16.494 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T12:01:16.494 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T12:01:16.494 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:01:16.494 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:16.494 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:01:16.494 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T12:01:16.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:01:16.495 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get fsid 2024-09-18T12:01:16.571 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T12:01:16.571 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:01:16.571 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T12:01:16.571 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T12:01:16.571 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T12:01:16.572 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T12:01:16.572 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T12:01:16.572 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T12:01:16.572 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:01:16.572 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:16.572 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:01:16.572 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76995/ceph-mon.a.asok 2024-09-18T12:01:16.572 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:01:16.572 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76995/ceph-mon.a.asok config get mon_host 2024-09-18T12:01:16.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1505: test_wait_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-09-18T12:01:16.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-18T12:01:16.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T12:01:16.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T12:01:16.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T12:01:16.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-18T12:01:16.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T12:01:17.495 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T12:01:17.495 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:01:17.495 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:01:17.495 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:01:17.496 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:17.496 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:01:17.496 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T12:01:17.496 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T12:01:17.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T12:01:17.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1506: test_wait_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-09-18T12:01:17.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-18T12:01:17.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:01:17.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T12:01:17.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:01:17.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-18T12:01:17.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-18T12:01:17.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:01:17.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:01:17.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:01:17.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-18T12:01:17.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-18T12:01:17.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:01:17.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T12:01:17.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-18T12:01:17.525 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:01:17.525 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:01:17.525 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:01:17.525 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:01:17.525 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:17.525 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:01:17.525 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T12:01:17.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' 2024-09-18T12:01:17.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:01:17.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:01:17.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:01:17.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-18T12:01:17.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-18T12:01:17.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:01:17.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:01:17.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:01:17.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:01:17.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:01:17.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T12:01:17.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-18T12:01:17.527 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:01:17.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b50a57ac-be98-45ee-831e-ec54b84e07b5 2024-09-18T12:01:17.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b50a57ac-be98-45ee-831e-ec54b84e07b5' 2024-09-18T12:01:17.527 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 b50a57ac-be98-45ee-831e-ec54b84e07b5 2024-09-18T12:01:17.527 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:01:17.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANwepmQ+wDIBAApmue0PolBN9u0z1FmIpNyg== 2024-09-18T12:01:17.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANwepmQ+wDIBAApmue0PolBN9u0z1FmIpNyg=="}' 2024-09-18T12:01:17.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b50a57ac-be98-45ee-831e-ec54b84e07b5 -i td/ceph-helpers/0/new.json 2024-09-18T12:01:17.847 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:01:17.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-18T12:01:17.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANwepmQ+wDIBAApmue0PolBN9u0z1FmIpNyg== --osd-uuid b50a57ac-be98-45ee-831e-ec54b84e07b5 2024-09-18T12:01:17.881 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:17.879+0000 7ff9bb08b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:01:17.902 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:17.891+0000 7ff9bb08b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:01:17.902 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:17.892+0000 7ff9bb08b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:01:17.902 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:17.892+0000 7ff9bb08b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T12:01:17.902 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:17.892+0000 7ff9bb08b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T12:01:17.902 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:17.892+0000 7ff9bb08b440 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-18T12:01:17.933 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:17.930+0000 7ff9bb08b440 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-18T12:01:23.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-18T12:01:23.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:01:23.291 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T12:01:23.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T12:01:23.292 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:01:23.774 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T12:01:23.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T12:01:23.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=92255036-ee37-4797-aa7f-f8e6b3f0325b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:01:23.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:01:23.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:01:23.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:01:23.797 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:23.795+0000 7f82a1d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:01:23.798 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:23.797+0000 7f82a1d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:01:23.799 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:23.798+0000 7f82a1d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:01:24.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T12:01:24.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:01:24.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T12:01:24.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:01:24.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:01:24.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:24.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:01:24.130 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:01:24.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:24.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:24.323 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:24.322+0000 7f82a1d23440 -1 Falling back to public interface 2024-09-18T12:01:24.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:25.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:25.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:25.487 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:01:25.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:01:25.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:25.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:25.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:26.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:26.851 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:01:26.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:26.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:01:26.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:26.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:26.878 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:26.874+0000 7f82a1d23440 -1 osd.0 0 log_to_monitors true 2024-09-18T12:01:27.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:28.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:28.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:28.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:01:28.258 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:01:28.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:28.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:28.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:29.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:29.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:29.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T12:01:29.701 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T12:01:29.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:29.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:30.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:31.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:31.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:31.086 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T12:01:31.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T12:01:31.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:31.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:31.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:32.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:32.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:32.561 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T12:01:32.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T12:01:32.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:32.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:32.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:33.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:33.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:33.938 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T12:01:33.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T12:01:33.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:33.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:34.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:35.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:35.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:35.352 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T12:01:35.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T12:01:35.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:35.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:35.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:36.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:36.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:36.774 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T12:01:36.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T12:01:36.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:36.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:37.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:38.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:38.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:38.152 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T12:01:38.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T12:01:38.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:38.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:38.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:39.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:39.582 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T12:01:39.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:39.583 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T12:01:39.583 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:39.583 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:40.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:41.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:41.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:41.527 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T12:01:41.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T12:01:41.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:41.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:42.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:43.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:43.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:43.363 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T12:01:43.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T12:01:43.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:43.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:44.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:45.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:45.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:45.412 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T12:01:45.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T12:01:45.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:45.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:45.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:46.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:46.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:46.881 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T12:01:46.881 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T12:01:46.881 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:46.881 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:47.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:48.359 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:48.359 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:48.360 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T12:01:48.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T12:01:48.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:48.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:49.090 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:01:50.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:01:50.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:01:50.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T12:01:50.092 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T12:01:50.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:01:50.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:01:50.476 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/172109,v1:127.0.0.1:6803/172109] [v2:127.0.0.1:6804/172109,v1:127.0.0.1:6805/172109] exists,up b50a57ac-be98-45ee-831e-ec54b84e07b5 2024-09-18T12:01:50.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:01:50.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:01:50.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:01:50.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1507: test_wait_osd_id_used_by_pgs: create_rbd_pool 2024-09-18T12:01:50.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T12:01:50.818 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' does not exist 2024-09-18T12:01:50.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T12:01:50.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T12:01:51.376 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'rbd' already exists 2024-09-18T12:01:51.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T12:01:52.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T12:01:53.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1508: test_wait_osd_id_used_by_pgs: wait_for_clean 2024-09-18T12:01:53.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T12:01:53.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T12:01:53.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T12:01:53.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T12:01:53.577 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T12:01:53.577 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T12:01:53.577 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T12:01:53.577 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T12:01:53.577 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T12:01:53.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T12:01:53.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T12:01:53.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T12:01:53.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T12:01:53.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T12:01:53.675 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T12:01:54.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T12:01:54.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T12:01:54.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:01:54.025 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T12:01:54.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836486 2024-09-18T12:01:54.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836486 2024-09-18T12:01:54.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836486' 2024-09-18T12:01:54.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:01:54.133 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836486 2024-09-18T12:01:54.133 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:01:54.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T12:01:54.134 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836486 2024-09-18T12:01:54.134 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:01:54.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836486 2024-09-18T12:01:54.135 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836486 2024-09-18T12:01:54.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-18T12:01:54.136 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T12:01:54.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-18T12:01:54.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T12:01:55.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T12:01:55.496 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T12:01:55.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836486 -lt 21474836486 2024-09-18T12:01:55.850 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T12:01:55.851 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T12:01:55.851 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T12:01:56.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T12:01:56.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T12:01:56.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T12:01:56.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T12:01:56.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T12:01:56.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T12:01:56.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T12:01:56.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T12:01:56.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T12:01:56.667 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T12:01:56.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T12:01:56.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T12:01:57.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T12:01:57.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T12:01:57.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T12:01:57.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1509: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 0 8 2024-09-18T12:01:57.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: local id=0 2024-09-18T12:01:57.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: local count=8 2024-09-18T12:01:57.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1487: wait_osd_id_used_by_pgs: status=1 2024-09-18T12:01:57.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i=0 )) 2024-09-18T12:01:57.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i < 300 / 5 )) 2024-09-18T12:01:57.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1489: wait_osd_id_used_by_pgs: echo 0 2024-09-18T12:01:57.150 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:01:57.151 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-09-18T12:01:57.151 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: grep -c 0 2024-09-18T12:01:57.151 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-09-18T12:01:57.151 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-09-18T12:01:57.151 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2024-09-18T12:01:57.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: test 8 = 8 2024-09-18T12:01:57.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1493: wait_osd_id_used_by_pgs: status=0 2024-09-18T12:01:57.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: wait_osd_id_used_by_pgs: break 2024-09-18T12:01:57.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: wait_osd_id_used_by_pgs: return 0 2024-09-18T12:01:57.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1510: test_wait_osd_id_used_by_pgs: TIMEOUT=1 2024-09-18T12:01:57.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1510: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 123 5 2024-09-18T12:01:57.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: local id=123 2024-09-18T12:01:57.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: local count=5 2024-09-18T12:01:57.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1487: wait_osd_id_used_by_pgs: status=1 2024-09-18T12:01:57.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i=0 )) 2024-09-18T12:01:57.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i < 1 / 5 )) 2024-09-18T12:01:57.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: wait_osd_id_used_by_pgs: return 1 2024-09-18T12:01:57.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1511: test_wait_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-09-18T12:01:57.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-18T12:01:57.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:01:57.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-18T12:01:57.595 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:01:57.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:01:57.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:01:57.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:01:57.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:01:57.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:01:57.699 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:01:57.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:01:57.700 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:01:57.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:01:57.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:01:57.701 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:01:57.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:01:57.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:01:57.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:01:57.703 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:01:57.703 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:01:57.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:01:57.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-18T12:01:57.710 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:01:57.710 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:57.710 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76995 2024-09-18T12:01:57.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76995 2024-09-18T12:01:57.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:01:57.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:01:57.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: : exit 0 2024-09-18T12:01:57.712 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T12:01:57.712 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T12:01:57.738 INFO:tasks.workunit:Running workunit misc/test-snaptrim-stats.sh... 2024-09-18T12:01:57.738 DEBUG:teuthology.orchestra.run.smithi157:workunit test misc/test-snaptrim-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh 2024-09-18T12:01:57.800 INFO:tasks.workunit.client.0.smithi157.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T12:01:57.805 INFO:tasks.workunit.client.0.smithi157.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T12:01:57.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T12:01:57.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T12:01:57.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T12:01:57.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T12:01:57.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T12:01:57.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T12:01:57.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T12:01:57.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T12:01:57.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/test-snaptrim-stats 2024-09-18T12:01:57.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:21: run: local dir=td/test-snaptrim-stats 2024-09-18T12:01:57.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:22: run: shift 2024-09-18T12:01:57.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-09-18T12:01:57.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-09-18T12:01:57.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:25: run: export CEPH_ARGS 2024-09-18T12:01:57.806 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: uuidgen 2024-09-18T12:01:57.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: CEPH_ARGS+='--fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none ' 2024-09-18T12:01:57.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-09-18T12:01:57.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:28: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-09-18T12:01:57.807 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: set 2024-09-18T12:01:57.807 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T12:01:57.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: local 'funcs=TEST_snaptrim_stats 2024-09-18T12:01:57.809 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_snaptrim_stats_multiple_snaps' 2024-09-18T12:01:57.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2024-09-18T12:01:57.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2024-09-18T12:01:57.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-09-18T12:01:57.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-09-18T12:01:57.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-18T12:01:57.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:01:57.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-18T12:01:57.810 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:01:57.810 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:01:57.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:01:57.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:01:57.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:01:57.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:01:57.811 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:01:57.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:01:57.811 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:01:57.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:01:57.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:01:57.812 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:01:57.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:01:57.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:01:57.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:01:57.813 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:01:57.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:01:57.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:01:57.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-18T12:01:57.816 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:01:57.816 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:57.816 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:01:57.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173406 2024-09-18T12:01:57.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:01:57.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:01:57.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-09-18T12:01:57.818 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T12:01:57.818 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:57.818 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:01:57.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173406 2024-09-18T12:01:57.821 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T12:01:57.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T12:01:57.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T12:01:57.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T12:01:57.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-09-18T12:01:57.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats td/test-snaptrim-stats 2024-09-18T12:01:57.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:39: TEST_snaptrim_stats: local dir=td/test-snaptrim-stats 2024-09-18T12:01:57.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:40: TEST_snaptrim_stats: local poolname=test 2024-09-18T12:01:57.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:41: TEST_snaptrim_stats: local OSDS=3 2024-09-18T12:01:57.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:42: TEST_snaptrim_stats: local PGNUM=8 2024-09-18T12:01:57.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:43: TEST_snaptrim_stats: local PGPNUM=8 2024-09-18T12:01:57.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:44: TEST_snaptrim_stats: local objects=10 2024-09-18T12:01:57.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:45: TEST_snaptrim_stats: local WAIT_FOR_UPDATE=10 2024-09-18T12:01:57.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:47: TEST_snaptrim_stats: setup td/test-snaptrim-stats 2024-09-18T12:01:57.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-09-18T12:01:57.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-09-18T12:01:57.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-18T12:01:57.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:01:57.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-18T12:01:57.822 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:01:57.822 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:01:57.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:01:57.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:01:57.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:01:57.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:01:57.823 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:01:57.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:01:57.824 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:01:57.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:01:57.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:01:57.824 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:01:57.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:01:57.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:01:57.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:01:57.825 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:01:57.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:01:57.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:01:57.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-18T12:01:57.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:01:57.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:57.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:01:57.828 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173406 2024-09-18T12:01:57.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:01:57.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:01:57.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-09-18T12:01:57.830 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T12:01:57.830 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:57.830 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:01:57.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173406 2024-09-18T12:01:57.832 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T12:01:57.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T12:01:57.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T12:01:57.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-09-18T12:01:57.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:48: TEST_snaptrim_stats: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2024-09-18T12:01:57.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-snaptrim-stats 2024-09-18T12:01:57.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T12:01:57.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T12:01:57.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T12:01:57.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-snaptrim-stats/a 2024-09-18T12:01:57.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2024-09-18T12:01:58.197 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T12:01:58.197 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:01:58.197 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:01:58.197 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:01:58.197 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:58.197 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:01:58.198 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173406/$cluster-$name.asok' 2024-09-18T12:01:58.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-18T12:01:58.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T12:01:58.441 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T12:01:58.441 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:01:58.441 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T12:01:58.441 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T12:01:58.441 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T12:01:58.441 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T12:01:58.441 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T12:01:58.441 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T12:01:58.441 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:01:58.442 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:58.442 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:01:58.442 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173406/ceph-mon.a.asok 2024-09-18T12:01:58.442 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:01:58.442 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.173406/ceph-mon.a.asok config get fsid 2024-09-18T12:01:58.515 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T12:01:58.515 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:01:58.515 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T12:01:58.515 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T12:01:58.515 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T12:01:58.515 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T12:01:58.515 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T12:01:58.516 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T12:01:58.516 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:01:58.516 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:58.516 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:01:58.516 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173406/ceph-mon.a.asok 2024-09-18T12:01:58.516 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:01:58.516 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.173406/ceph-mon.a.asok config get mon_host 2024-09-18T12:01:58.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:49: TEST_snaptrim_stats: run_mgr td/test-snaptrim-stats x 2024-09-18T12:01:58.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-snaptrim-stats 2024-09-18T12:01:58.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T12:01:58.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T12:01:58.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T12:01:58.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-snaptrim-stats/x 2024-09-18T12:01:58.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T12:01:59.426 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T12:01:59.426 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:01:59.426 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:01:59.426 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:01:59.427 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:59.427 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:01:59.427 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173406/$cluster-$name.asok' 2024-09-18T12:01:59.427 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T12:01:59.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T12:01:59.450 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: expr 3 - 1 2024-09-18T12:01:59.451 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: seq 0 2 2024-09-18T12:01:59.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T12:01:59.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:01:59.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-18T12:01:59.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:01:59.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T12:01:59.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:01:59.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/0 2024-09-18T12:01:59.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-18T12:01:59.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:01:59.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:01:59.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:01:59.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2024-09-18T12:01:59.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2024-09-18T12:01:59.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:01:59.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T12:01:59.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-18T12:01:59.456 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:01:59.456 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:01:59.456 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:01:59.456 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:01:59.456 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:01:59.456 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:01:59.456 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173406/$cluster-$name.asok' 2024-09-18T12:01:59.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' 2024-09-18T12:01:59.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:01:59.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:01:59.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:01:59.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-18T12:01:59.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-18T12:01:59.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:01:59.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:01:59.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:01:59.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:01:59.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:01:59.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:01:59.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/0 2024-09-18T12:01:59.457 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:01:59.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c691bb3c-70ff-410e-ad28-f2df0773573f 2024-09-18T12:01:59.458 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c691bb3c-70ff-410e-ad28-f2df0773573f' 2024-09-18T12:01:59.458 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 c691bb3c-70ff-410e-ad28-f2df0773573f 2024-09-18T12:01:59.458 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:01:59.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3wepmQX/UGxAAZmKBX7F7j1L39bmko3pT2Q== 2024-09-18T12:01:59.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3wepmQX/UGxAAZmKBX7F7j1L39bmko3pT2Q=="}' 2024-09-18T12:01:59.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c691bb3c-70ff-410e-ad28-f2df0773573f -i td/test-snaptrim-stats/0/new.json 2024-09-18T12:01:59.771 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:01:59.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/0/new.json 2024-09-18T12:01:59.865 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQA3wepmQX/UGxAAZmKBX7F7j1L39bmko3pT2Q== --osd-uuid c691bb3c-70ff-410e-ad28-f2df0773573f 2024-09-18T12:01:59.865 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:59.803+0000 7f669bb2a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:01:59.865 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:59.815+0000 7f669bb2a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:01:59.865 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:59.816+0000 7f669bb2a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:01:59.865 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:59.816+0000 7f669bb2a440 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-18T12:01:59.865 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:59.817+0000 7f669bb2a440 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-18T12:01:59.865 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:59.817+0000 7f669bb2a440 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-18T12:01:59.865 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:01:59.847+0000 7f669bb2a440 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2024-09-18T12:02:05.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2024-09-18T12:02:05.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:02:05.554 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T12:02:05.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T12:02:05.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:02:06.050 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T12:02:06.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T12:02:06.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:02:06.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:02:06.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:02:06.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:02:06.071 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:06.070+0000 7f839931f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:02:06.073 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:06.072+0000 7f839931f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:02:06.074 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:06.073+0000 7f839931f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:02:06.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T12:02:06.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:02:06.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T12:02:06.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:02:06.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:02:06.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:06.399 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:02:06.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:02:06.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:06.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:06.590 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:06.589+0000 7f839931f440 -1 Falling back to public interface 2024-09-18T12:02:06.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:07.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:07.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:07.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:02:07.754 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:02:07.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:07.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:08.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:09.092 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:09.091+0000 7f839931f440 -1 osd.0 0 log_to_monitors true 2024-09-18T12:02:09.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:09.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:09.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:02:09.111 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:02:09.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:09.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:09.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:10.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:10.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:10.507 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:02:10.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:02:10.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:10.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:10.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:11.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:11.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:11.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T12:02:11.964 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T12:02:11.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:11.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:12.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:13.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:13.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:13.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T12:02:13.361 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T12:02:13.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:13.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:13.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:14.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:14.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:14.837 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T12:02:14.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T12:02:14.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:14.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:15.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:16.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:16.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:16.198 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T12:02:16.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T12:02:16.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:16.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:16.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:17.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:17.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:17.560 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T12:02:17.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T12:02:17.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:17.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:17.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:18.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:18.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:18.966 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T12:02:18.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T12:02:18.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:18.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:19.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:20.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:20.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:20.472 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T12:02:20.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T12:02:20.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:20.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:20.916 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:21.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:21.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:21.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T12:02:21.918 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T12:02:21.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:21.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:22.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:23.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:23.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:23.887 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T12:02:23.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T12:02:23.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:23.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:24.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:25.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:25.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:25.695 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T12:02:25.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T12:02:25.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:25.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:26.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:27.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:27.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:27.883 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T12:02:27.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T12:02:27.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:27.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:28.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:29.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:29.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:29.624 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T12:02:29.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T12:02:29.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:29.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:30.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:31.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:31.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:31.776 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T12:02:31.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T12:02:31.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:31.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:32.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:33.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:33.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:33.285 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T12:02:33.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T12:02:33.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:33.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:02:33.665 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/174516,v1:127.0.0.1:6803/174516] [v2:127.0.0.1:6804/174516,v1:127.0.0.1:6805/174516] exists,up c691bb3c-70ff-410e-ad28-f2df0773573f 2024-09-18T12:02:33.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:02:33.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:02:33.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:02:33.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T12:02:33.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:02:33.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-18T12:02:33.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:02:33.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T12:02:33.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:02:33.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/1 2024-09-18T12:02:33.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-18T12:02:33.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:02:33.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:02:33.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:02:33.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2024-09-18T12:02:33.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2024-09-18T12:02:33.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:02:33.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T12:02:33.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-18T12:02:33.670 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:02:33.670 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:02:33.670 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:02:33.670 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:02:33.670 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:02:33.671 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:02:33.671 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173406/$cluster-$name.asok' 2024-09-18T12:02:33.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' 2024-09-18T12:02:33.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:02:33.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:02:33.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:02:33.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-18T12:02:33.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-18T12:02:33.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:02:33.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:02:33.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:02:33.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:02:33.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:02:33.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:02:33.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/1 2024-09-18T12:02:33.672 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:02:33.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=389a0849-b9f2-4272-b67d-56fb622d8fd3 2024-09-18T12:02:33.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 389a0849-b9f2-4272-b67d-56fb622d8fd3' 2024-09-18T12:02:33.672 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 389a0849-b9f2-4272-b67d-56fb622d8fd3 2024-09-18T12:02:33.672 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:02:33.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBZwepmXVqKKBAApjutbQFzExphy9Ton6JxEg== 2024-09-18T12:02:33.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBZwepmXVqKKBAApjutbQFzExphy9Ton6JxEg=="}' 2024-09-18T12:02:33.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 389a0849-b9f2-4272-b67d-56fb622d8fd3 -i td/test-snaptrim-stats/1/new.json 2024-09-18T12:02:34.138 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:02:34.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/1/new.json 2024-09-18T12:02:34.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQBZwepmXVqKKBAApjutbQFzExphy9Ton6JxEg== --osd-uuid 389a0849-b9f2-4272-b67d-56fb622d8fd3 2024-09-18T12:02:34.174 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:34.172+0000 7fd32a57b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:02:34.176 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:34.174+0000 7fd32a57b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:02:34.176 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:34.175+0000 7fd32a57b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:02:34.177 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:34.175+0000 7fd32a57b440 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-18T12:02:34.177 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:34.175+0000 7fd32a57b440 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-18T12:02:34.177 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:34.175+0000 7fd32a57b440 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-18T12:02:34.220 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:34.219+0000 7fd32a57b440 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2024-09-18T12:02:40.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2024-09-18T12:02:40.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:02:40.142 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T12:02:40.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T12:02:40.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:02:40.629 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T12:02:40.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T12:02:40.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:02:40.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:02:40.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:02:40.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:02:40.651 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:40.649+0000 7f643d37a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:02:40.653 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:40.651+0000 7f643d37a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:02:40.654 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:40.652+0000 7f643d37a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:02:40.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T12:02:40.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:02:40.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T12:02:40.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:02:40.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:02:40.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:40.978 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:02:40.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:02:40.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:40.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:02:41.182 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:41.181+0000 7f643d37a440 -1 Falling back to public interface 2024-09-18T12:02:41.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:42.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:42.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:42.337 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:02:42.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:02:42.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:42.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:02:42.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:43.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:43.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:43.685 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:02:43.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:02:43.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:43.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:02:43.859 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:02:43.857+0000 7f643d37a440 -1 osd.1 0 log_to_monitors true 2024-09-18T12:02:44.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:45.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:45.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:45.067 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:02:45.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:02:45.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:45.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:02:45.463 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:46.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:46.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:46.464 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T12:02:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T12:02:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:02:46.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:47.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:47.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:47.876 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T12:02:47.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T12:02:47.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:47.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:02:48.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:49.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:49.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:49.251 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T12:02:49.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T12:02:49.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:49.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:02:49.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:50.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:50.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:50.719 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T12:02:50.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T12:02:50.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:50.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:02:51.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:52.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:52.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:52.076 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T12:02:52.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T12:02:52.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:52.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:02:52.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:53.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:53.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:53.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T12:02:53.485 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T12:02:53.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:53.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:02:54.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:55.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:55.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:55.050 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T12:02:55.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T12:02:55.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:55.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:02:55.408 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:56.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:56.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:56.410 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T12:02:56.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T12:02:56.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:56.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:02:56.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:57.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:57.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:57.943 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T12:02:57.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T12:02:57.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:57.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:02:58.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:02:59.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:02:59.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:02:59.813 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T12:02:59.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T12:02:59.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:02:59.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:03:00.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:01.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:01.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:01.794 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T12:03:01.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T12:03:01.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:01.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:03:02.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:03.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:03.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:03.626 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T12:03:03.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T12:03:03.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:03.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:03:04.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:05.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:05.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:05.435 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T12:03:05.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T12:03:05.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:05.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:03:06.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:07.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:07.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:07.331 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T12:03:07.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T12:03:07.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:07.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:03:08.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:09.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:09.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:09.232 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T12:03:09.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T12:03:09.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:09.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:03:09.631 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/176270,v1:127.0.0.1:6811/176270] [v2:127.0.0.1:6812/176270,v1:127.0.0.1:6813/176270] exists,up 389a0849-b9f2-4272-b67d-56fb622d8fd3 2024-09-18T12:03:09.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:03:09.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:03:09.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:03:09.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T12:03:09.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:03:09.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-18T12:03:09.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:03:09.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T12:03:09.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:03:09.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/2 2024-09-18T12:03:09.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-18T12:03:09.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:03:09.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:03:09.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:03:09.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2024-09-18T12:03:09.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2024-09-18T12:03:09.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:03:09.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T12:03:09.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-18T12:03:09.636 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:03:09.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:03:09.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:03:09.637 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:03:09.637 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:03:09.637 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:03:09.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173406/$cluster-$name.asok' 2024-09-18T12:03:09.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' 2024-09-18T12:03:09.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:03:09.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:03:09.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:03:09.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-18T12:03:09.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-18T12:03:09.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:03:09.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:03:09.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:03:09.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:03:09.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:03:09.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:03:09.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/2 2024-09-18T12:03:09.638 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:03:09.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fe85846b-19a6-4480-a3a9-d1765f3f81ea 2024-09-18T12:03:09.638 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 fe85846b-19a6-4480-a3a9-d1765f3f81ea 2024-09-18T12:03:09.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fe85846b-19a6-4480-a3a9-d1765f3f81ea' 2024-09-18T12:03:09.639 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:03:09.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9wepm8DGcJhAA014ehb0BBCuzPetU0liHQg== 2024-09-18T12:03:09.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9wepm8DGcJhAA014ehb0BBCuzPetU0liHQg=="}' 2024-09-18T12:03:09.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fe85846b-19a6-4480-a3a9-d1765f3f81ea -i td/test-snaptrim-stats/2/new.json 2024-09-18T12:03:10.139 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:03:10.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/2/new.json 2024-09-18T12:03:10.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQB9wepm8DGcJhAA014ehb0BBCuzPetU0liHQg== --osd-uuid fe85846b-19a6-4480-a3a9-d1765f3f81ea 2024-09-18T12:03:10.174 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:03:10.173+0000 7ffb17417440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:03:10.176 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:03:10.175+0000 7ffb17417440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:03:10.177 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:03:10.176+0000 7ffb17417440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:03:10.177 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:03:10.176+0000 7ffb17417440 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-18T12:03:10.177 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:03:10.176+0000 7ffb17417440 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-18T12:03:10.178 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:03:10.176+0000 7ffb17417440 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-18T12:03:10.254 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:03:10.253+0000 7ffb17417440 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2024-09-18T12:03:16.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2024-09-18T12:03:16.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:03:16.279 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-18T12:03:16.280 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T12:03:16.280 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:03:16.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T12:03:16.763 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-18T12:03:16.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:03:16.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:03:16.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:03:16.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:03:16.784 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:03:16.782+0000 7f82b5a73440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:03:16.786 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:03:16.784+0000 7f82b5a73440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:03:16.787 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:03:16.785+0000 7f82b5a73440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:03:17.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T12:03:17.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:03:17.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T12:03:17.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:03:17.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:03:17.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:17.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:03:17.111 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:03:17.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:17.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:17.304 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:03:17.303+0000 7f82b5a73440 -1 Falling back to public interface 2024-09-18T12:03:17.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:18.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:18.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:18.461 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:03:18.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:03:18.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:18.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:18.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:19.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:19.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:19.811 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:03:19.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:03:19.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:19.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:19.901 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:03:19.899+0000 7f82b5a73440 -1 osd.2 0 log_to_monitors true 2024-09-18T12:03:20.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:21.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:21.218 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:03:21.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:21.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:03:21.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:21.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:21.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:22.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:22.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:22.693 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T12:03:22.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T12:03:22.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:22.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:23.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:24.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:24.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:24.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T12:03:24.072 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T12:03:24.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:24.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:24.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:25.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:25.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:25.453 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T12:03:25.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T12:03:25.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:25.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:25.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:26.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:26.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:26.826 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T12:03:26.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T12:03:26.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:26.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:27.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:28.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:28.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:28.232 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T12:03:28.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T12:03:28.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:28.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:28.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:29.724 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:29.724 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:29.724 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T12:03:29.725 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T12:03:29.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:29.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:30.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:31.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:31.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:31.213 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T12:03:31.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T12:03:31.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:31.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:31.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:32.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:32.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:32.631 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T12:03:32.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T12:03:32.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:32.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:33.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:34.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:34.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:34.348 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T12:03:34.349 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T12:03:34.349 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:34.349 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:35.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:36.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:36.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:36.176 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T12:03:36.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T12:03:36.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:36.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:37.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:38.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:38.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:38.010 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T12:03:38.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T12:03:38.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:38.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:38.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:39.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:39.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:39.899 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T12:03:39.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T12:03:39.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:39.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:40.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:41.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:41.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:41.734 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T12:03:41.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T12:03:41.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:41.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:42.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:43.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:43.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:43.771 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T12:03:43.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T12:03:43.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:43.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:44.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:03:45.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:03:45.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:03:45.177 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T12:03:45.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T12:03:45.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:03:45.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:03:45.525 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/178049,v1:127.0.0.1:6819/178049] [v2:127.0.0.1:6820/178049,v1:127.0.0.1:6821/178049] exists,up fe85846b-19a6-4480-a3a9-d1765f3f81ea 2024-09-18T12:03:45.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:03:45.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:03:45.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:03:45.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:56: TEST_snaptrim_stats: ceph osd set noscrub 2024-09-18T12:03:46.188 INFO:tasks.workunit.client.0.smithi157.stderr:noscrub is set 2024-09-18T12:03:46.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:57: TEST_snaptrim_stats: ceph osd set nodeep-scrub 2024-09-18T12:03:47.058 INFO:tasks.workunit.client.0.smithi157.stderr:nodeep-scrub is set 2024-09-18T12:03:47.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:60: TEST_snaptrim_stats: create_pool test 8 8 2024-09-18T12:03:47.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 8 8 2024-09-18T12:03:47.779 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'test' already exists 2024-09-18T12:03:47.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T12:03:48.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:61: TEST_snaptrim_stats: wait_for_clean 2024-09-18T12:03:48.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T12:03:48.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T12:03:48.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T12:03:48.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T12:03:48.793 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T12:03:48.793 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T12:03:48.793 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T12:03:48.793 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T12:03:48.793 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T12:03:48.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T12:03:48.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T12:03:48.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T12:03:48.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T12:03:48.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T12:03:48.884 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T12:03:49.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T12:03:49.238 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-18T12:03:49.239 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-18T12:03:49.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T12:03:49.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:03:49.239 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T12:03:49.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836501 2024-09-18T12:03:49.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836501 2024-09-18T12:03:49.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501' 2024-09-18T12:03:49.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:03:49.407 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T12:03:49.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672972 2024-09-18T12:03:49.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672972 2024-09-18T12:03:49.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501 1-42949672972' 2024-09-18T12:03:49.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:03:49.588 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T12:03:49.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509450 2024-09-18T12:03:49.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509450 2024-09-18T12:03:49.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501 1-42949672972 2-64424509450' 2024-09-18T12:03:49.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:03:49.693 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836501 2024-09-18T12:03:49.693 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:03:49.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T12:03:49.694 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836501 2024-09-18T12:03:49.695 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:03:49.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836501 2024-09-18T12:03:49.696 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836501 2024-09-18T12:03:49.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-18T12:03:49.696 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T12:03:50.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836499 -lt 21474836501 2024-09-18T12:03:50.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T12:03:51.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T12:03:51.115 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T12:03:51.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-18T12:03:51.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:03:51.472 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672972 2024-09-18T12:03:51.472 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:03:51.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T12:03:51.473 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672972 2024-09-18T12:03:51.473 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:03:51.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672972 2024-09-18T12:03:51.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-18T12:03:51.475 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672972 2024-09-18T12:03:51.475 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T12:03:51.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-18T12:03:51.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:03:51.826 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509450 2024-09-18T12:03:51.827 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:03:51.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T12:03:51.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509450 2024-09-18T12:03:51.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:03:51.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509450 2024-09-18T12:03:51.829 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-18T12:03:51.829 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509450 2024-09-18T12:03:51.830 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T12:03:52.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-18T12:03:52.176 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T12:03:52.176 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T12:03:52.176 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T12:03:52.648 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-18T12:03:52.648 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T12:03:52.648 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T12:03:52.648 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T12:03:52.648 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T12:03:52.648 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T12:03:52.648 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T12:03:52.648 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T12:03:52.985 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T12:03:52.986 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T12:03:52.986 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T12:03:52.986 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T12:03:53.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-18T12:03:53.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T12:03:53.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T12:03:53.467 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: ceph osd dump 2024-09-18T12:03:53.468 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: grep '^pool.*['\'']test['\'']' 2024-09-18T12:03:53.468 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: awk '{ print $2 }' 2024-09-18T12:03:53.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: poolid=1 2024-09-18T12:03:53.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:65: TEST_snaptrim_stats: TESTDATA=testdata.1 2024-09-18T12:03:53.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:66: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2024-09-18T12:03:53.907 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records in 2024-09-18T12:03:53.907 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records out 2024-09-18T12:03:53.907 INFO:tasks.workunit.client.0.smithi157.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 4.7844e-05 s, 85.6 MB/s 2024-09-18T12:03:53.908 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: seq 1 10 2024-09-18T12:03:53.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:53.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj1 testdata.1 2024-09-18T12:03:54.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:54.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj2 testdata.1 2024-09-18T12:03:54.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:54.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj3 testdata.1 2024-09-18T12:03:54.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:54.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj4 testdata.1 2024-09-18T12:03:54.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:54.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj5 testdata.1 2024-09-18T12:03:54.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:54.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj6 testdata.1 2024-09-18T12:03:54.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:54.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj7 testdata.1 2024-09-18T12:03:55.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:55.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj8 testdata.1 2024-09-18T12:03:55.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:55.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj9 testdata.1 2024-09-18T12:03:55.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:55.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj10 testdata.1 2024-09-18T12:03:55.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:71: TEST_snaptrim_stats: rm -f testdata.1 2024-09-18T12:03:55.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:74: TEST_snaptrim_stats: SNAP=1 2024-09-18T12:03:55.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:75: TEST_snaptrim_stats: rados -p test mksnap snap1 2024-09-18T12:03:55.684 INFO:tasks.workunit.client.0.smithi157.stdout:created pool test snap snap1 2024-09-18T12:03:55.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:76: TEST_snaptrim_stats: TESTDATA=testdata.2 2024-09-18T12:03:55.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:77: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2024-09-18T12:03:55.687 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records in 2024-09-18T12:03:55.687 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records out 2024-09-18T12:03:55.687 INFO:tasks.workunit.client.0.smithi157.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 5.0733e-05 s, 80.7 MB/s 2024-09-18T12:03:55.687 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: seq 1 10 2024-09-18T12:03:55.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:55.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj1 testdata.2 2024-09-18T12:03:56.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:56.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj2 testdata.2 2024-09-18T12:03:56.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:56.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj3 testdata.2 2024-09-18T12:03:56.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:56.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj4 testdata.2 2024-09-18T12:03:56.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:56.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj5 testdata.2 2024-09-18T12:03:56.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:56.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj6 testdata.2 2024-09-18T12:03:56.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:56.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj7 testdata.2 2024-09-18T12:03:56.985 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:56.985 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj8 testdata.2 2024-09-18T12:03:57.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:57.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj9 testdata.2 2024-09-18T12:03:57.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-18T12:03:57.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj10 testdata.2 2024-09-18T12:03:57.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:82: TEST_snaptrim_stats: rm -f testdata.2 2024-09-18T12:03:57.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:85: TEST_snaptrim_stats: rados -p test rmsnap snap1 2024-09-18T12:03:57.628 INFO:tasks.workunit.client.0.smithi157.stdout:removed pool test snap snap1 2024-09-18T12:03:57.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:88: TEST_snaptrim_stats: wait_for_clean 2024-09-18T12:03:57.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T12:03:57.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T12:03:57.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T12:03:57.630 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T12:03:57.631 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T12:03:57.631 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T12:03:57.631 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T12:03:57.631 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T12:03:57.631 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T12:03:57.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T12:03:57.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T12:03:57.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T12:03:57.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T12:03:57.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T12:03:57.717 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T12:03:58.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T12:03:58.082 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-18T12:03:58.082 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-18T12:03:58.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T12:03:58.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:03:58.082 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T12:03:58.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836504 2024-09-18T12:03:58.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836504 2024-09-18T12:03:58.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504' 2024-09-18T12:03:58.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:03:58.187 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T12:03:58.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672975 2024-09-18T12:03:58.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672975 2024-09-18T12:03:58.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-42949672975' 2024-09-18T12:03:58.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:03:58.333 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T12:03:58.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509453 2024-09-18T12:03:58.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509453 2024-09-18T12:03:58.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-42949672975 2-64424509453' 2024-09-18T12:03:58.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:03:58.437 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836504 2024-09-18T12:03:58.437 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:03:58.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T12:03:58.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836504 2024-09-18T12:03:58.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:03:58.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836504 2024-09-18T12:03:58.440 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836504 2024-09-18T12:03:58.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-18T12:03:58.440 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T12:03:58.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-18T12:03:58.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T12:03:59.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T12:03:59.793 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T12:04:00.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836504 2024-09-18T12:04:00.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:04:00.148 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672975 2024-09-18T12:04:00.148 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:04:00.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T12:04:00.149 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672975 2024-09-18T12:04:00.149 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:04:00.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672975 2024-09-18T12:04:00.151 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672975 2024-09-18T12:04:00.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-18T12:04:00.151 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T12:04:00.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-18T12:04:00.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:04:00.512 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509453 2024-09-18T12:04:00.512 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:04:00.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T12:04:00.514 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509453 2024-09-18T12:04:00.514 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:04:00.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509453 2024-09-18T12:04:00.515 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509453 2024-09-18T12:04:00.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-18T12:04:00.515 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T12:04:00.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-18T12:04:00.859 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T12:04:00.859 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T12:04:00.859 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T12:04:01.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-18T12:04:01.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T12:04:01.331 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T12:04:01.331 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T12:04:01.331 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T12:04:01.331 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T12:04:01.331 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T12:04:01.331 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T12:04:01.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T12:04:01.674 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T12:04:01.674 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T12:04:01.675 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T12:04:02.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-18T12:04:02.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T12:04:02.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T12:04:02.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:89: TEST_snaptrim_stats: sleep 10 2024-09-18T12:04:12.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:90: TEST_snaptrim_stats: local objects_trimmed=0 2024-09-18T12:04:12.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:91: TEST_snaptrim_stats: local snaptrim_duration_total=0.0 2024-09-18T12:04:12.146 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: expr 8 - 1 2024-09-18T12:04:12.147 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: seq 0 7 2024-09-18T12:04:12.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:04:12.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.0 2024-09-18T12:04:12.150 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.0 query 2024-09-18T12:04:12.150 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-18T12:04:12.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 0 + 2 2024-09-18T12:04:12.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=2 2024-09-18T12:04:12.272 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-18T12:04:12.272 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.0 query 2024-09-18T12:04:12.273 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-18T12:04:12.382 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 0.0 + 0.79316403000000002 2024-09-18T12:04:12.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.79316403000000002 2024-09-18T12:04:12.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:04:12.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.1 2024-09-18T12:04:12.384 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.1 query 2024-09-18T12:04:12.384 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-18T12:04:12.492 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 2 + 2 2024-09-18T12:04:12.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2024-09-18T12:04:12.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-18T12:04:12.494 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.1 query 2024-09-18T12:04:12.494 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-18T12:04:12.602 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .79316403000000002 + 0.32669546599999999 2024-09-18T12:04:12.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.11985949600000001 2024-09-18T12:04:12.603 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:04:12.603 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.2 2024-09-18T12:04:12.603 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.2 query 2024-09-18T12:04:12.603 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-18T12:04:12.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 0 2024-09-18T12:04:12.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2024-09-18T12:04:12.714 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-18T12:04:12.715 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.2 query 2024-09-18T12:04:12.715 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-18T12:04:12.823 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.11985949600000001 + 0.000241513 2024-09-18T12:04:12.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.12010100900000001 2024-09-18T12:04:12.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:04:12.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.3 2024-09-18T12:04:12.825 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.3 query 2024-09-18T12:04:12.825 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-18T12:04:12.932 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 2 2024-09-18T12:04:12.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=6 2024-09-18T12:04:12.934 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-18T12:04:12.935 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.3 query 2024-09-18T12:04:12.935 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-18T12:04:13.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.12010100900000001 + 0.43061942399999997 2024-09-18T12:04:13.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.55072043299999998 2024-09-18T12:04:13.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:04:13.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.4 2024-09-18T12:04:13.045 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.4 query 2024-09-18T12:04:13.045 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-18T12:04:13.154 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 6 + 1 2024-09-18T12:04:13.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=7 2024-09-18T12:04:13.155 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-18T12:04:13.156 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.4 query 2024-09-18T12:04:13.156 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-18T12:04:13.265 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.55072043299999998 + 0.29359097899999997 2024-09-18T12:04:13.265 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.84431141199999995 2024-09-18T12:04:13.265 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:04:13.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.5 2024-09-18T12:04:13.266 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.5 query 2024-09-18T12:04:13.266 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-18T12:04:13.375 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 7 + 1 2024-09-18T12:04:13.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2024-09-18T12:04:13.377 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-18T12:04:13.377 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.5 query 2024-09-18T12:04:13.377 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-18T12:04:13.485 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.84431141199999995 + 0.17591916199999999 2024-09-18T12:04:13.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=2.02023057399999994 2024-09-18T12:04:13.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:04:13.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.6 2024-09-18T12:04:13.487 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.6 query 2024-09-18T12:04:13.487 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-18T12:04:13.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 0 2024-09-18T12:04:13.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2024-09-18T12:04:13.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-18T12:04:13.599 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.6 query 2024-09-18T12:04:13.599 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-18T12:04:13.708 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 2.02023057399999994 + 0.293797327 2024-09-18T12:04:13.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=2.31402790099999994 2024-09-18T12:04:13.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:04:13.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.7 2024-09-18T12:04:13.709 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.7 query 2024-09-18T12:04:13.709 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-18T12:04:13.817 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 2 2024-09-18T12:04:13.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=10 2024-09-18T12:04:13.819 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-18T12:04:13.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.7 query 2024-09-18T12:04:13.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-18T12:04:13.929 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 2.31402790099999994 + 0.96407074000000004 2024-09-18T12:04:13.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=3.27809864099999998 2024-09-18T12:04:13.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:100: TEST_snaptrim_stats: test 10 -eq 10 2024-09-18T12:04:13.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: echo '3.27809864099999998 > 0.0' 2024-09-18T12:04:13.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: bc 2024-09-18T12:04:13.930 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:04:13.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:103: TEST_snaptrim_stats: teardown td/test-snaptrim-stats 2024-09-18T12:04:13.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-18T12:04:13.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:04:13.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-18T12:04:13.931 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:04:13.931 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:04:13.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:04:13.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:04:13.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:04:14.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:04:14.039 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:04:14.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:04:14.040 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:04:14.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:04:14.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:04:14.041 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:04:14.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:04:14.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:04:14.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:04:14.042 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:04:14.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:04:14.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:04:14.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-18T12:04:14.074 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:04:14.075 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:04:14.075 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:04:14.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173406 2024-09-18T12:04:14.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:04:14.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:04:14.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2024-09-18T12:04:14.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-18T12:04:14.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:04:14.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-18T12:04:14.077 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:04:14.077 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:04:14.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:04:14.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:04:14.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:04:14.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:04:14.078 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:04:14.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:04:14.079 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:04:14.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:04:14.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:04:14.080 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:04:14.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:04:14.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:04:14.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:04:14.081 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:04:14.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:04:14.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:04:14.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-18T12:04:14.086 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:04:14.086 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:04:14.086 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:04:14.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173406 2024-09-18T12:04:14.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:04:14.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:04:14.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2024-09-18T12:04:14.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2024-09-18T12:04:14.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-09-18T12:04:14.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-09-18T12:04:14.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-18T12:04:14.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:04:14.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-18T12:04:14.087 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:04:14.087 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:04:14.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:04:14.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:04:14.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:04:14.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:04:14.088 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:04:14.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:04:14.089 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:04:14.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:04:14.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:04:14.089 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:04:14.090 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:04:14.090 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:04:14.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:04:14.091 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:04:14.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:04:14.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:04:14.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-18T12:04:14.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:04:14.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:04:14.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:04:14.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173406 2024-09-18T12:04:14.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:04:14.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:04:14.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-09-18T12:04:14.096 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T12:04:14.096 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:04:14.096 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:04:14.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173406 2024-09-18T12:04:14.099 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T12:04:14.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T12:04:14.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T12:04:14.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-09-18T12:04:14.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats_multiple_snaps td/test-snaptrim-stats 2024-09-18T12:04:14.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:107: TEST_snaptrim_stats_multiple_snaps: local dir=td/test-snaptrim-stats 2024-09-18T12:04:14.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:108: TEST_snaptrim_stats_multiple_snaps: local poolname=test 2024-09-18T12:04:14.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:109: TEST_snaptrim_stats_multiple_snaps: local OSDS=3 2024-09-18T12:04:14.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:110: TEST_snaptrim_stats_multiple_snaps: local PGNUM=8 2024-09-18T12:04:14.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:111: TEST_snaptrim_stats_multiple_snaps: local PGPNUM=8 2024-09-18T12:04:14.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:112: TEST_snaptrim_stats_multiple_snaps: local objects=10 2024-09-18T12:04:14.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:113: TEST_snaptrim_stats_multiple_snaps: local WAIT_FOR_UPDATE=10 2024-09-18T12:04:14.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:115: TEST_snaptrim_stats_multiple_snaps: setup td/test-snaptrim-stats 2024-09-18T12:04:14.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-09-18T12:04:14.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-09-18T12:04:14.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-18T12:04:14.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:04:14.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-18T12:04:14.100 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:04:14.100 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:04:14.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:04:14.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:04:14.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:04:14.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:04:14.101 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:04:14.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:04:14.102 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:04:14.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:04:14.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:04:14.102 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:04:14.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:04:14.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:04:14.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:04:14.103 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:04:14.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:04:14.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:04:14.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-18T12:04:14.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:04:14.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:04:14.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:04:14.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173406 2024-09-18T12:04:14.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:04:14.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:04:14.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-09-18T12:04:14.108 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T12:04:14.108 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:04:14.108 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:04:14.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173406 2024-09-18T12:04:14.110 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T12:04:14.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T12:04:14.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T12:04:14.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-09-18T12:04:14.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:116: TEST_snaptrim_stats_multiple_snaps: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2024-09-18T12:04:14.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-snaptrim-stats 2024-09-18T12:04:14.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T12:04:14.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T12:04:14.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T12:04:14.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-snaptrim-stats/a 2024-09-18T12:04:14.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2024-09-18T12:04:14.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T12:04:14.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:04:14.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:04:14.494 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:04:14.495 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:04:14.495 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:04:14.495 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173406/$cluster-$name.asok' 2024-09-18T12:04:14.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-18T12:04:14.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T12:04:14.746 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T12:04:14.746 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:04:14.746 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T12:04:14.746 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T12:04:14.746 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T12:04:14.747 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T12:04:14.747 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T12:04:14.747 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T12:04:14.748 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:04:14.748 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:04:14.748 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:04:14.748 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173406/ceph-mon.a.asok 2024-09-18T12:04:14.748 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:04:14.748 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.173406/ceph-mon.a.asok config get fsid 2024-09-18T12:04:14.823 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T12:04:14.823 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:04:14.823 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T12:04:14.823 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T12:04:14.823 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T12:04:14.823 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T12:04:14.823 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T12:04:14.823 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T12:04:14.824 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:04:14.824 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:04:14.824 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:04:14.824 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173406/ceph-mon.a.asok 2024-09-18T12:04:14.824 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:04:14.824 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.173406/ceph-mon.a.asok config get mon_host 2024-09-18T12:04:14.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:117: TEST_snaptrim_stats_multiple_snaps: run_mgr td/test-snaptrim-stats x 2024-09-18T12:04:14.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-snaptrim-stats 2024-09-18T12:04:14.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T12:04:14.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T12:04:14.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T12:04:14.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-snaptrim-stats/x 2024-09-18T12:04:14.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T12:04:15.732 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T12:04:15.732 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:04:15.732 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:04:15.732 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:04:15.732 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:04:15.732 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:04:15.732 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173406/$cluster-$name.asok' 2024-09-18T12:04:15.733 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T12:04:15.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T12:04:15.755 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: expr 3 - 1 2024-09-18T12:04:15.756 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: seq 0 2 2024-09-18T12:04:15.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T12:04:15.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:04:15.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-18T12:04:15.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:04:15.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T12:04:15.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:04:15.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/0 2024-09-18T12:04:15.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-18T12:04:15.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:04:15.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:04:15.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:04:15.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2024-09-18T12:04:15.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2024-09-18T12:04:15.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:04:15.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T12:04:15.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-18T12:04:15.762 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:04:15.762 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:04:15.762 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:04:15.762 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:04:15.762 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:04:15.762 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:04:15.762 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173406/$cluster-$name.asok' 2024-09-18T12:04:15.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' 2024-09-18T12:04:15.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:04:15.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:04:15.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:04:15.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-18T12:04:15.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-18T12:04:15.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:04:15.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:04:15.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:04:15.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:04:15.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:04:15.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:04:15.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/0 2024-09-18T12:04:15.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:04:15.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=50ff9e3a-be96-4c59-ae84-8fc2e9da8bd9 2024-09-18T12:04:15.764 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 50ff9e3a-be96-4c59-ae84-8fc2e9da8bd9 2024-09-18T12:04:15.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 50ff9e3a-be96-4c59-ae84-8fc2e9da8bd9' 2024-09-18T12:04:15.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:04:15.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/wepmW0oNLhAABIrWV4DlRCiB9wY7gyLyZA== 2024-09-18T12:04:15.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/wepmW0oNLhAABIrWV4DlRCiB9wY7gyLyZA=="}' 2024-09-18T12:04:15.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 50ff9e3a-be96-4c59-ae84-8fc2e9da8bd9 -i td/test-snaptrim-stats/0/new.json 2024-09-18T12:04:16.083 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:04:16.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/0/new.json 2024-09-18T12:04:16.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQC/wepmW0oNLhAABIrWV4DlRCiB9wY7gyLyZA== --osd-uuid 50ff9e3a-be96-4c59-ae84-8fc2e9da8bd9 2024-09-18T12:04:16.118 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:16.116+0000 7f6a52b1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:04:16.130 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:16.129+0000 7f6a52b1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:04:16.131 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:16.130+0000 7f6a52b1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:04:16.132 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:16.131+0000 7f6a52b1c440 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-18T12:04:16.132 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:16.131+0000 7f6a52b1c440 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-18T12:04:16.132 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:16.131+0000 7f6a52b1c440 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-18T12:04:16.170 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:16.168+0000 7f6a52b1c440 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2024-09-18T12:04:21.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2024-09-18T12:04:21.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:04:21.744 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T12:04:21.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T12:04:21.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:04:22.228 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T12:04:22.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T12:04:22.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:04:22.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:04:22.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:04:22.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:04:22.250 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:22.248+0000 7fbe8e20b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:04:22.252 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:22.250+0000 7fbe8e20b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:04:22.253 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:22.251+0000 7fbe8e20b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:04:22.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T12:04:22.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:04:22.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T12:04:22.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:04:22.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:04:22.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:22.575 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:04:22.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:04:22.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:22.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:22.779 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:22.778+0000 7fbe8e20b440 -1 Falling back to public interface 2024-09-18T12:04:22.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:23.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:23.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:23.948 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:04:23.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:04:23.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:23.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:24.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:25.303 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:25.301+0000 7fbe8e20b440 -1 osd.0 0 log_to_monitors true 2024-09-18T12:04:25.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:25.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:25.306 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:04:25.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:04:25.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:25.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:25.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:26.724 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:26.724 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:26.725 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:04:26.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:04:26.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:26.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:27.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:28.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:28.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:28.191 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T12:04:28.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T12:04:28.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:28.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:28.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:29.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:29.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:29.579 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T12:04:29.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T12:04:29.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:29.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:30.019 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:31.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:31.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:31.022 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T12:04:31.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T12:04:31.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:31.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:31.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:32.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:32.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:32.414 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T12:04:32.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T12:04:32.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:32.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:32.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:33.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:33.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:33.778 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T12:04:33.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T12:04:33.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:33.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:34.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:35.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:35.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:35.158 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T12:04:35.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T12:04:35.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:35.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:35.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:36.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:36.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:36.542 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T12:04:36.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T12:04:36.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:36.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:37.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:38.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:38.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:38.011 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T12:04:38.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T12:04:38.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:38.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:38.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:39.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:39.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:39.729 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T12:04:39.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T12:04:39.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:39.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:40.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:41.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:41.639 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T12:04:41.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:41.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T12:04:41.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:41.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:42.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:43.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:43.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:43.400 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T12:04:43.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T12:04:43.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:43.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:44.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:45.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:45.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:45.467 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T12:04:45.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T12:04:45.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:45.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:46.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:47.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:47.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:47.387 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T12:04:47.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T12:04:47.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:47.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:47.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:48.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:48.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:48.856 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T12:04:48.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T12:04:48.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:48.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:04:49.214 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/181780,v1:127.0.0.1:6803/181780] [v2:127.0.0.1:6804/181780,v1:127.0.0.1:6805/181780] exists,up 50ff9e3a-be96-4c59-ae84-8fc2e9da8bd9 2024-09-18T12:04:49.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:04:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:04:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:04:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T12:04:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:04:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-18T12:04:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:04:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T12:04:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:04:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/1 2024-09-18T12:04:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-18T12:04:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:04:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:04:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:04:49.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2024-09-18T12:04:49.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2024-09-18T12:04:49.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:04:49.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T12:04:49.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-18T12:04:49.220 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:04:49.220 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:04:49.220 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:04:49.220 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:04:49.220 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:04:49.220 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:04:49.220 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173406/$cluster-$name.asok' 2024-09-18T12:04:49.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' 2024-09-18T12:04:49.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:04:49.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:04:49.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:04:49.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-18T12:04:49.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-18T12:04:49.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:04:49.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:04:49.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:04:49.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:04:49.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:04:49.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:04:49.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/1 2024-09-18T12:04:49.221 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:04:49.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=be002343-c656-4b02-9ed3-2dc06671f23b 2024-09-18T12:04:49.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 be002343-c656-4b02-9ed3-2dc06671f23b' 2024-09-18T12:04:49.222 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 be002343-c656-4b02-9ed3-2dc06671f23b 2024-09-18T12:04:49.222 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:04:49.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhwepmAH26DRAAb6t9jb0hEprxlb5qthvpRA== 2024-09-18T12:04:49.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhwepmAH26DRAAb6t9jb0hEprxlb5qthvpRA=="}' 2024-09-18T12:04:49.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new be002343-c656-4b02-9ed3-2dc06671f23b -i td/test-snaptrim-stats/1/new.json 2024-09-18T12:04:49.710 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:04:49.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/1/new.json 2024-09-18T12:04:49.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQDhwepmAH26DRAAb6t9jb0hEprxlb5qthvpRA== --osd-uuid be002343-c656-4b02-9ed3-2dc06671f23b 2024-09-18T12:04:49.745 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:49.744+0000 7fd3fc721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:04:49.747 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:49.746+0000 7fd3fc721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:04:49.748 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:49.747+0000 7fd3fc721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:04:49.748 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:49.747+0000 7fd3fc721440 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-18T12:04:49.749 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:49.747+0000 7fd3fc721440 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-18T12:04:49.749 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:49.747+0000 7fd3fc721440 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-18T12:04:49.937 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:49.937+0000 7fd3fc721440 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2024-09-18T12:04:56.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2024-09-18T12:04:56.724 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:04:56.724 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T12:04:56.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T12:04:56.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:04:57.205 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T12:04:57.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T12:04:57.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:04:57.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:04:57.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:04:57.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:04:57.226 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:57.224+0000 7fbaede3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:04:57.228 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:57.226+0000 7fbaede3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:04:57.229 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:57.227+0000 7fbaede3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:04:57.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T12:04:57.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:04:57.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T12:04:57.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:04:57.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:04:57.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:57.581 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:04:57.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:04:57.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:57.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:04:57.745 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:04:57.743+0000 7fbaede3b440 -1 Falling back to public interface 2024-09-18T12:04:57.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:04:58.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:04:58.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:04:58.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:04:58.938 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:04:58.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:04:58.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:04:59.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:00.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:00.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:00.297 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:05:00.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:05:00.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:00.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:05:00.312 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:05:00.310+0000 7fbaede3b440 -1 osd.1 0 log_to_monitors true 2024-09-18T12:05:00.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:01.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:01.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:01.781 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:05:01.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:05:01.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:01.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:05:02.268 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:03.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:03.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:03.270 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T12:05:03.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T12:05:03.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:03.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:05:03.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:04.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:04.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:04.665 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T12:05:04.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T12:05:04.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:04.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:05:05.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:06.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:06.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:06.116 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T12:05:06.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T12:05:06.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:06.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:05:06.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:07.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:07.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:07.504 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T12:05:07.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T12:05:07.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:07.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:05:07.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:08.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:08.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:08.868 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T12:05:08.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T12:05:08.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:08.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:05:09.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:10.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:10.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:10.261 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T12:05:10.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T12:05:10.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:10.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:05:10.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:11.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:11.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:11.622 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T12:05:11.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T12:05:11.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:11.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:05:12.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:13.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:13.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:13.061 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T12:05:13.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T12:05:13.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:13.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:05:13.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:14.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:14.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:14.921 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T12:05:14.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T12:05:14.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:14.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:05:15.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:16.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:16.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:16.868 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T12:05:16.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T12:05:16.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:16.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:05:17.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:18.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:18.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:18.751 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T12:05:18.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T12:05:18.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:18.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:05:19.535 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:20.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:20.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:20.537 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T12:05:20.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T12:05:20.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:20.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:05:21.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:22.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:22.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:22.716 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T12:05:22.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T12:05:22.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:22.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:05:23.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:24.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:24.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:24.601 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T12:05:24.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T12:05:24.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:24.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:05:24.955 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/183534,v1:127.0.0.1:6811/183534] [v2:127.0.0.1:6812/183534,v1:127.0.0.1:6813/183534] exists,up be002343-c656-4b02-9ed3-2dc06671f23b 2024-09-18T12:05:24.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:05:24.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:05:24.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:05:24.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T12:05:24.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:05:24.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-18T12:05:24.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:05:24.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T12:05:24.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:05:24.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/2 2024-09-18T12:05:24.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-18T12:05:24.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:05:24.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:05:24.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:05:24.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2024-09-18T12:05:24.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2024-09-18T12:05:24.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:05:24.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T12:05:24.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-18T12:05:24.961 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:05:24.961 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:05:24.961 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:05:24.961 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:05:24.961 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:05:24.961 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:05:24.961 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173406/$cluster-$name.asok' 2024-09-18T12:05:24.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' 2024-09-18T12:05:24.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:05:24.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:05:24.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:05:24.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-18T12:05:24.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-18T12:05:24.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:05:24.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:05:24.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:05:24.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:05:24.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:05:24.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:05:24.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/2 2024-09-18T12:05:24.962 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:05:24.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=59e89562-5f4a-4a3a-bd45-6f12f26b3f9a 2024-09-18T12:05:24.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 59e89562-5f4a-4a3a-bd45-6f12f26b3f9a' 2024-09-18T12:05:24.963 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 59e89562-5f4a-4a3a-bd45-6f12f26b3f9a 2024-09-18T12:05:24.963 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:05:24.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAEwupmAC3fORAAzvOtOObluOrhgcqJZlUWLw== 2024-09-18T12:05:24.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAEwupmAC3fORAAzvOtOObluOrhgcqJZlUWLw=="}' 2024-09-18T12:05:24.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 59e89562-5f4a-4a3a-bd45-6f12f26b3f9a -i td/test-snaptrim-stats/2/new.json 2024-09-18T12:05:25.493 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:05:25.500 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/2/new.json 2024-09-18T12:05:25.501 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQAEwupmAC3fORAAzvOtOObluOrhgcqJZlUWLw== --osd-uuid 59e89562-5f4a-4a3a-bd45-6f12f26b3f9a 2024-09-18T12:05:25.525 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:05:25.523+0000 7f27ac06e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:05:25.526 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:05:25.525+0000 7f27ac06e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:05:25.527 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:05:25.526+0000 7f27ac06e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:05:25.528 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:05:25.526+0000 7f27ac06e440 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-18T12:05:25.528 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:05:25.526+0000 7f27ac06e440 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-18T12:05:25.528 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:05:25.526+0000 7f27ac06e440 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-18T12:05:25.605 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:05:25.604+0000 7f27ac06e440 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2024-09-18T12:05:31.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2024-09-18T12:05:31.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:05:31.444 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-18T12:05:31.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T12:05:31.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:05:31.936 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-18T12:05:31.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T12:05:31.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=24ecbec1-6a01-4e3d-b5b8-66c8bcfb286d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.173406/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-18T12:05:31.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:05:31.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:05:31.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:05:31.957 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:05:31.956+0000 7f8260867440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:05:31.959 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:05:31.958+0000 7f8260867440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:05:31.960 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:05:31.959+0000 7f8260867440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:05:32.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T12:05:32.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:05:32.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T12:05:32.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:05:32.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:05:32.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:32.298 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:05:32.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:05:32.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:32.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:32.477 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:05:32.476+0000 7f8260867440 -1 Falling back to public interface 2024-09-18T12:05:32.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:33.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:33.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:33.682 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:05:33.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:05:33.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:33.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:34.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:34.988 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:05:34.987+0000 7f8260867440 -1 osd.2 0 log_to_monitors true 2024-09-18T12:05:35.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:35.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:35.076 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:05:35.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:05:35.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:35.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:35.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:36.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:36.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:36.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:05:36.623 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:05:36.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:36.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:37.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:38.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:38.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:38.104 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T12:05:38.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T12:05:38.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:38.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:38.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:39.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:39.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:39.538 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T12:05:39.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T12:05:39.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:39.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:39.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:40.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:40.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:40.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T12:05:40.927 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T12:05:40.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:40.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:41.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:42.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:42.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:42.322 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T12:05:42.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T12:05:42.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:42.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:42.824 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:43.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:43.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:43.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T12:05:43.826 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T12:05:43.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:43.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:44.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:45.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:45.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:45.331 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T12:05:45.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T12:05:45.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:45.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:45.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:46.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:46.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:46.806 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T12:05:46.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T12:05:46.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:46.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:47.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:48.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:48.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:48.194 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T12:05:48.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T12:05:48.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:48.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:49.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:50.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:50.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:50.105 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T12:05:50.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T12:05:50.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:50.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:51.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:52.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:52.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:52.072 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T12:05:52.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T12:05:52.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:52.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:53.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:54.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:54.210 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T12:05:54.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:54.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T12:05:54.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:54.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:55.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:56.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:56.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:56.086 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T12:05:56.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T12:05:56.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:56.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:56.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:05:57.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:05:57.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:05:57.934 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T12:05:57.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T12:05:57.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:05:57.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:05:59.035 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:06:00.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:06:00.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:06:00.037 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T12:06:00.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T12:06:00.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:06:00.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:06:00.387 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/185288,v1:127.0.0.1:6819/185288] [v2:127.0.0.1:6820/185288,v1:127.0.0.1:6821/185288] exists,up 59e89562-5f4a-4a3a-bd45-6f12f26b3f9a 2024-09-18T12:06:00.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:06:00.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:06:00.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:06:00.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:124: TEST_snaptrim_stats_multiple_snaps: ceph osd set noscrub 2024-09-18T12:06:01.020 INFO:tasks.workunit.client.0.smithi157.stderr:noscrub is set 2024-09-18T12:06:01.031 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:125: TEST_snaptrim_stats_multiple_snaps: ceph osd set nodeep-scrub 2024-09-18T12:06:01.783 INFO:tasks.workunit.client.0.smithi157.stderr:nodeep-scrub is set 2024-09-18T12:06:01.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:128: TEST_snaptrim_stats_multiple_snaps: create_pool test 8 8 2024-09-18T12:06:01.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 8 8 2024-09-18T12:06:02.443 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'test' already exists 2024-09-18T12:06:02.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T12:06:03.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:129: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-09-18T12:06:03.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T12:06:03.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T12:06:03.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T12:06:03.456 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T12:06:03.456 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T12:06:03.456 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T12:06:03.457 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T12:06:03.457 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T12:06:03.457 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T12:06:03.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T12:06:03.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T12:06:03.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T12:06:03.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T12:06:03.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T12:06:03.552 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T12:06:03.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T12:06:03.939 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-18T12:06:03.939 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-18T12:06:03.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T12:06:03.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:06:03.939 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T12:06:04.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836505 2024-09-18T12:06:04.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836505 2024-09-18T12:06:04.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505' 2024-09-18T12:06:04.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:06:04.071 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T12:06:04.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672975 2024-09-18T12:06:04.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672975 2024-09-18T12:06:04.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672975' 2024-09-18T12:06:04.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:06:04.175 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T12:06:04.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509447 2024-09-18T12:06:04.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509447 2024-09-18T12:06:04.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672975 2-64424509447' 2024-09-18T12:06:04.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:06:04.307 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836505 2024-09-18T12:06:04.307 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:06:04.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T12:06:04.308 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836505 2024-09-18T12:06:04.308 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:06:04.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836505 2024-09-18T12:06:04.309 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836505 2024-09-18T12:06:04.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-18T12:06:04.309 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T12:06:04.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836503 -lt 21474836505 2024-09-18T12:06:04.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T12:06:05.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T12:06:05.658 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T12:06:06.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836506 -lt 21474836505 2024-09-18T12:06:06.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:06:06.037 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672975 2024-09-18T12:06:06.037 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:06:06.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T12:06:06.038 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672975 2024-09-18T12:06:06.038 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:06:06.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672975 2024-09-18T12:06:06.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-18T12:06:06.039 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672975 2024-09-18T12:06:06.039 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T12:06:06.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672976 -lt 42949672975 2024-09-18T12:06:06.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:06:06.387 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509447 2024-09-18T12:06:06.387 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:06:06.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T12:06:06.388 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509447 2024-09-18T12:06:06.388 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:06:06.389 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509447 2024-09-18T12:06:06.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509447 2024-09-18T12:06:06.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-18T12:06:06.389 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T12:06:06.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-18T12:06:06.773 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T12:06:06.773 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T12:06:06.773 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T12:06:07.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-18T12:06:07.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T12:06:07.253 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T12:06:07.253 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T12:06:07.253 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T12:06:07.253 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T12:06:07.253 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T12:06:07.254 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T12:06:07.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T12:06:07.591 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T12:06:07.591 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T12:06:07.591 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T12:06:08.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-18T12:06:08.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T12:06:08.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T12:06:08.085 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: ceph osd dump 2024-09-18T12:06:08.085 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: grep '^pool.*['\'']test['\'']' 2024-09-18T12:06:08.085 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: awk '{ print $2 }' 2024-09-18T12:06:08.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: poolid=1 2024-09-18T12:06:08.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:133: TEST_snaptrim_stats_multiple_snaps: local TESTDATA=testdata.0 2024-09-18T12:06:08.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:134: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.0 bs=4096 count=1 2024-09-18T12:06:08.431 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records in 2024-09-18T12:06:08.431 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records out 2024-09-18T12:06:08.431 INFO:tasks.workunit.client.0.smithi157.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 5.5996e-05 s, 73.1 MB/s 2024-09-18T12:06:08.431 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-09-18T12:06:08.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:08.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.0 2024-09-18T12:06:08.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:08.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.0 2024-09-18T12:06:08.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:08.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.0 2024-09-18T12:06:08.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:08.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.0 2024-09-18T12:06:09.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:09.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.0 2024-09-18T12:06:09.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:09.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.0 2024-09-18T12:06:09.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:09.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.0 2024-09-18T12:06:09.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:09.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.0 2024-09-18T12:06:09.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:09.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.0 2024-09-18T12:06:09.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:09.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.0 2024-09-18T12:06:09.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:139: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.0 2024-09-18T12:06:09.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:142: TEST_snaptrim_stats_multiple_snaps: NUMSNAPS=2 2024-09-18T12:06:09.919 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2024-09-18T12:06:09.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-09-18T12:06:09.920 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap1 2024-09-18T12:06:10.093 INFO:tasks.workunit.client.0.smithi157.stdout:created pool test snap snap1 2024-09-18T12:06:10.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.1 2024-09-18T12:06:10.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2024-09-18T12:06:10.095 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records in 2024-09-18T12:06:10.095 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records out 2024-09-18T12:06:10.095 INFO:tasks.workunit.client.0.smithi157.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 4.7968e-05 s, 85.4 MB/s 2024-09-18T12:06:10.096 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-09-18T12:06:10.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:10.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.1 2024-09-18T12:06:10.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:10.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.1 2024-09-18T12:06:10.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:10.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.1 2024-09-18T12:06:10.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:10.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.1 2024-09-18T12:06:10.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:10.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.1 2024-09-18T12:06:10.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:10.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.1 2024-09-18T12:06:11.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:11.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.1 2024-09-18T12:06:11.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:11.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.1 2024-09-18T12:06:11.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:11.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.1 2024-09-18T12:06:11.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:11.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.1 2024-09-18T12:06:11.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.1 2024-09-18T12:06:11.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-09-18T12:06:11.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap2 2024-09-18T12:06:11.927 INFO:tasks.workunit.client.0.smithi157.stdout:created pool test snap snap2 2024-09-18T12:06:11.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.2 2024-09-18T12:06:11.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2024-09-18T12:06:11.930 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records in 2024-09-18T12:06:11.930 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records out 2024-09-18T12:06:11.930 INFO:tasks.workunit.client.0.smithi157.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 5.156e-05 s, 79.4 MB/s 2024-09-18T12:06:11.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-09-18T12:06:11.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:11.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.2 2024-09-18T12:06:12.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:12.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.2 2024-09-18T12:06:12.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:12.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.2 2024-09-18T12:06:12.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:12.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.2 2024-09-18T12:06:12.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:12.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.2 2024-09-18T12:06:12.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:12.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.2 2024-09-18T12:06:12.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:12.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.2 2024-09-18T12:06:13.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:13.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.2 2024-09-18T12:06:13.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:13.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.2 2024-09-18T12:06:13.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-18T12:06:13.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.2 2024-09-18T12:06:13.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.2 2024-09-18T12:06:13.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:156: TEST_snaptrim_stats_multiple_snaps: local total_objects_trimmed=0 2024-09-18T12:06:13.515 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2024-09-18T12:06:13.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-09-18T12:06:13.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap1 2024-09-18T12:06:13.664 INFO:tasks.workunit.client.0.smithi157.stdout:removed pool test snap snap1 2024-09-18T12:06:13.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-09-18T12:06:13.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T12:06:13.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T12:06:13.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T12:06:13.666 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T12:06:13.667 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T12:06:13.667 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T12:06:13.667 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T12:06:13.667 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T12:06:13.667 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T12:06:13.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T12:06:13.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T12:06:13.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T12:06:13.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T12:06:13.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T12:06:13.754 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T12:06:14.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T12:06:14.110 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-18T12:06:14.110 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-18T12:06:14.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T12:06:14.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:06:14.111 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T12:06:14.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836509 2024-09-18T12:06:14.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836509 2024-09-18T12:06:14.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509' 2024-09-18T12:06:14.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:06:14.327 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T12:06:14.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672979 2024-09-18T12:06:14.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672979 2024-09-18T12:06:14.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672979' 2024-09-18T12:06:14.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:06:14.434 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T12:06:14.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509451 2024-09-18T12:06:14.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509451 2024-09-18T12:06:14.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672979 2-64424509451' 2024-09-18T12:06:14.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:06:14.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836509 2024-09-18T12:06:14.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:06:14.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T12:06:14.630 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836509 2024-09-18T12:06:14.631 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:06:14.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836509 2024-09-18T12:06:14.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-18T12:06:14.632 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836509 2024-09-18T12:06:14.632 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T12:06:14.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836509 2024-09-18T12:06:14.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T12:06:15.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T12:06:15.992 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T12:06:16.349 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836510 -lt 21474836509 2024-09-18T12:06:16.349 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:06:16.349 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672979 2024-09-18T12:06:16.350 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:06:16.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T12:06:16.351 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672979 2024-09-18T12:06:16.351 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:06:16.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672979 2024-09-18T12:06:16.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-18T12:06:16.352 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672979 2024-09-18T12:06:16.352 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T12:06:16.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672980 -lt 42949672979 2024-09-18T12:06:16.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:06:16.702 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509451 2024-09-18T12:06:16.702 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:06:16.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T12:06:16.703 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509451 2024-09-18T12:06:16.703 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:06:16.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509451 2024-09-18T12:06:16.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-18T12:06:16.705 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509451 2024-09-18T12:06:16.705 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T12:06:17.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-18T12:06:17.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T12:06:17.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T12:06:17.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T12:06:17.583 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-18T12:06:17.583 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T12:06:17.583 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T12:06:17.583 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T12:06:17.583 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T12:06:17.583 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T12:06:17.584 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T12:06:17.584 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T12:06:17.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T12:06:17.928 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T12:06:17.928 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T12:06:17.928 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T12:06:18.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-18T12:06:18.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T12:06:18.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T12:06:18.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2024-09-18T12:06:28.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2024-09-18T12:06:28.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2024-09-18T12:06:28.395 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2024-09-18T12:06:28.396 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2024-09-18T12:06:28.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:06:28.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2024-09-18T12:06:28.398 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-09-18T12:06:28.398 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-18T12:06:28.517 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2024-09-18T12:06:28.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2024-09-18T12:06:28.518 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:28.519 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-09-18T12:06:28.519 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-18T12:06:28.628 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 0.550002251 2024-09-18T12:06:28.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.550002251 2024-09-18T12:06:28.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:06:28.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2024-09-18T12:06:28.629 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-09-18T12:06:28.629 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-18T12:06:28.739 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2024-09-18T12:06:28.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-09-18T12:06:28.740 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:28.741 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-09-18T12:06:28.741 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-18T12:06:28.849 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .550002251 + 0.40269301200000002 2024-09-18T12:06:28.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.95269526300000002 2024-09-18T12:06:28.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:06:28.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2024-09-18T12:06:28.850 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-09-18T12:06:28.850 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-18T12:06:28.959 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2024-09-18T12:06:28.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-09-18T12:06:28.960 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:28.961 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-09-18T12:06:28.961 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-18T12:06:29.070 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .95269526300000002 + 0.000306626 2024-09-18T12:06:29.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.95300188900000002 2024-09-18T12:06:29.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:06:29.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2024-09-18T12:06:29.071 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-09-18T12:06:29.071 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-18T12:06:29.181 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2024-09-18T12:06:29.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2024-09-18T12:06:29.182 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:29.182 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-09-18T12:06:29.183 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-18T12:06:29.292 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .95300188900000002 + 1.011667965 2024-09-18T12:06:29.293 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.96466985400000002 2024-09-18T12:06:29.293 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:06:29.293 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2024-09-18T12:06:29.293 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-09-18T12:06:29.293 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-18T12:06:29.402 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2024-09-18T12:06:29.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2024-09-18T12:06:29.404 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:29.404 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-09-18T12:06:29.404 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-18T12:06:29.513 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.96466985400000002 + 0.38235054499999999 2024-09-18T12:06:29.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.34702039900000001 2024-09-18T12:06:29.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:06:29.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2024-09-18T12:06:29.515 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-09-18T12:06:29.515 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-18T12:06:29.623 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2024-09-18T12:06:29.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-09-18T12:06:29.625 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:29.625 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-09-18T12:06:29.625 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-18T12:06:29.735 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.34702039900000001 + 0.40303239600000001 2024-09-18T12:06:29.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.75005279500000002 2024-09-18T12:06:29.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:06:29.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2024-09-18T12:06:29.736 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-09-18T12:06:29.736 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-18T12:06:29.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2024-09-18T12:06:29.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-09-18T12:06:29.846 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:29.847 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-09-18T12:06:29.847 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-18T12:06:29.964 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.75005279500000002 + 0.88548830499999998 2024-09-18T12:06:29.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=3.63554110000000000 2024-09-18T12:06:29.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:06:29.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2024-09-18T12:06:29.965 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-09-18T12:06:29.965 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-18T12:06:30.074 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2024-09-18T12:06:30.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2024-09-18T12:06:30.075 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:30.075 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-09-18T12:06:30.075 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-18T12:06:30.184 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 3.63554110000000000 + 0.88578703000000003 2024-09-18T12:06:30.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=4.52132813000000003 2024-09-18T12:06:30.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2024-09-18T12:06:30.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '4.52132813000000003 > 0.0' 2024-09-18T12:06:30.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:30.185 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:06:30.186 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 0 + 10 2024-09-18T12:06:30.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=10 2024-09-18T12:06:30.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-09-18T12:06:30.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap2 2024-09-18T12:06:30.334 INFO:tasks.workunit.client.0.smithi157.stdout:removed pool test snap snap2 2024-09-18T12:06:30.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-09-18T12:06:30.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T12:06:30.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T12:06:30.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T12:06:30.336 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T12:06:30.337 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T12:06:30.337 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T12:06:30.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T12:06:30.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T12:06:30.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T12:06:30.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T12:06:30.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T12:06:30.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T12:06:30.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T12:06:30.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T12:06:30.424 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T12:06:30.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T12:06:30.790 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-18T12:06:30.790 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-18T12:06:30.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T12:06:30.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:06:30.790 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T12:06:30.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836515 2024-09-18T12:06:30.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836515 2024-09-18T12:06:30.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836515' 2024-09-18T12:06:30.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:06:30.894 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T12:06:30.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672985 2024-09-18T12:06:30.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672985 2024-09-18T12:06:30.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836515 1-42949672985' 2024-09-18T12:06:30.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T12:06:30.999 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T12:06:31.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509457 2024-09-18T12:06:31.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509457 2024-09-18T12:06:31.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836515 1-42949672985 2-64424509457' 2024-09-18T12:06:31.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:06:31.155 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836515 2024-09-18T12:06:31.155 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:06:31.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T12:06:31.156 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836515 2024-09-18T12:06:31.156 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:06:31.157 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836515 2024-09-18T12:06:31.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836515 2024-09-18T12:06:31.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-09-18T12:06:31.157 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T12:06:31.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836515 -lt 21474836515 2024-09-18T12:06:31.603 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:06:31.603 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672985 2024-09-18T12:06:31.604 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:06:31.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T12:06:31.604 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672985 2024-09-18T12:06:31.604 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:06:31.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672985 2024-09-18T12:06:31.604 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672985 2024-09-18T12:06:31.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-18T12:06:31.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T12:06:32.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672985 -lt 42949672985 2024-09-18T12:06:32.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T12:06:32.070 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509457 2024-09-18T12:06:32.070 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T12:06:32.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T12:06:32.072 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509457 2024-09-18T12:06:32.072 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T12:06:32.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509457 2024-09-18T12:06:32.073 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509457 2024-09-18T12:06:32.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-18T12:06:32.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T12:06:32.498 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-18T12:06:32.499 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T12:06:32.499 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T12:06:32.499 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T12:06:32.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-18T12:06:32.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T12:06:32.979 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T12:06:32.979 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T12:06:32.979 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T12:06:32.979 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T12:06:32.979 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T12:06:32.979 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T12:06:33.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T12:06:33.321 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T12:06:33.321 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T12:06:33.321 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T12:06:33.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-18T12:06:33.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T12:06:33.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T12:06:33.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2024-09-18T12:06:43.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2024-09-18T12:06:43.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2024-09-18T12:06:43.795 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2024-09-18T12:06:43.797 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2024-09-18T12:06:43.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:06:43.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2024-09-18T12:06:43.799 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-09-18T12:06:43.799 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-18T12:06:43.917 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2024-09-18T12:06:43.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2024-09-18T12:06:43.919 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:43.919 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-09-18T12:06:43.919 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-18T12:06:44.028 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 0.60059724699999995 2024-09-18T12:06:44.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.60059724699999995 2024-09-18T12:06:44.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:06:44.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2024-09-18T12:06:44.030 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-09-18T12:06:44.030 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-18T12:06:44.138 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2024-09-18T12:06:44.139 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-09-18T12:06:44.140 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:44.140 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-09-18T12:06:44.141 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-18T12:06:44.250 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .60059724699999995 + 0.34370988499999999 2024-09-18T12:06:44.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.94430713199999994 2024-09-18T12:06:44.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:06:44.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2024-09-18T12:06:44.251 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-09-18T12:06:44.251 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-18T12:06:44.361 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2024-09-18T12:06:44.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-09-18T12:06:44.362 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:44.363 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-09-18T12:06:44.363 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-18T12:06:44.471 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .94430713199999994 + 0.00035328300000000003 2024-09-18T12:06:44.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.94466041499999994003 2024-09-18T12:06:44.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:06:44.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2024-09-18T12:06:44.472 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-09-18T12:06:44.472 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-18T12:06:44.580 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2024-09-18T12:06:44.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2024-09-18T12:06:44.582 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:44.582 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-09-18T12:06:44.582 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-18T12:06:44.690 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .94466041499999994003 + 0.306808415 2024-09-18T12:06:44.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.25146882999999994003 2024-09-18T12:06:44.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:06:44.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2024-09-18T12:06:44.692 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-09-18T12:06:44.692 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-18T12:06:44.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2024-09-18T12:06:44.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2024-09-18T12:06:44.802 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:44.802 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-09-18T12:06:44.803 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-18T12:06:44.910 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.25146882999999994003 + 0.27338285499999998 2024-09-18T12:06:44.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.52485168499999992003 2024-09-18T12:06:44.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:06:44.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2024-09-18T12:06:44.912 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-09-18T12:06:44.912 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-18T12:06:45.020 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2024-09-18T12:06:45.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-09-18T12:06:45.021 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:45.022 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-09-18T12:06:45.022 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-18T12:06:45.130 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.52485168499999992003 + 0.244095325 2024-09-18T12:06:45.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.76894700999999992003 2024-09-18T12:06:45.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:06:45.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2024-09-18T12:06:45.132 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-09-18T12:06:45.132 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-18T12:06:45.240 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2024-09-18T12:06:45.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-09-18T12:06:45.241 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:45.242 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-09-18T12:06:45.242 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-18T12:06:45.351 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.76894700999999992003 + 0.27362188199999998 2024-09-18T12:06:45.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.04256889199999990003 2024-09-18T12:06:45.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-18T12:06:45.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2024-09-18T12:06:45.353 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-09-18T12:06:45.353 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-18T12:06:45.462 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2024-09-18T12:06:45.463 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2024-09-18T12:06:45.464 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:45.464 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-09-18T12:06:45.465 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-18T12:06:45.572 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.04256889199999990003 + 0.79369847000000004 2024-09-18T12:06:45.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.83626736199999994003 2024-09-18T12:06:45.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2024-09-18T12:06:45.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '2.83626736199999994003 > 0.0' 2024-09-18T12:06:45.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-18T12:06:45.574 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:06:45.574 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 10 + 10 2024-09-18T12:06:45.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=20 2024-09-18T12:06:45.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:179: TEST_snaptrim_stats_multiple_snaps: test 20 -eq 20 2024-09-18T12:06:45.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:181: TEST_snaptrim_stats_multiple_snaps: teardown td/test-snaptrim-stats 2024-09-18T12:06:45.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-18T12:06:45.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:06:45.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-18T12:06:45.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:06:45.577 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:06:45.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:06:45.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:06:45.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:06:45.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:06:45.683 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:06:45.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:06:45.684 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:06:45.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:06:45.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:06:45.685 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:06:45.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:06:45.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:06:45.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:06:45.687 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:06:45.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:06:45.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:06:45.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-18T12:06:45.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:06:45.721 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:06:45.721 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:06:45.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173406 2024-09-18T12:06:45.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:06:45.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:06:45.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2024-09-18T12:06:45.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-18T12:06:45.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:06:45.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-18T12:06:45.723 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:06:45.723 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:06:45.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:06:45.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:06:45.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:06:45.724 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:06:45.724 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:06:45.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:06:45.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:06:45.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:06:45.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:06:45.726 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:06:45.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:06:45.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:06:45.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:06:45.728 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:06:45.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:06:45.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:06:45.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-18T12:06:45.731 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:06:45.731 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:06:45.731 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:06:45.731 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173406 2024-09-18T12:06:45.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:06:45.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:06:45.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T12:06:45.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/test-snaptrim-stats 0 2024-09-18T12:06:45.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-18T12:06:45.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T12:06:45.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-18T12:06:45.733 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:06:45.733 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:06:45.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:06:45.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:06:45.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:06:45.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:06:45.735 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:06:45.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:06:45.735 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:06:45.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:06:45.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:06:45.737 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:06:45.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:06:45.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:06:45.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:06:45.738 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:06:45.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:06:45.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T12:06:45.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-18T12:06:45.740 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:06:45.740 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:06:45.740 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173406 2024-09-18T12:06:45.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173406 2024-09-18T12:06:45.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:06:45.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:06:45.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T12:06:45.742 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T12:06:45.743 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T12:06:45.767 INFO:tasks.workunit:Running workunit misc/ver-health.sh... 2024-09-18T12:06:45.768 DEBUG:teuthology.orchestra.run.smithi157:workunit test misc/ver-health.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh 2024-09-18T12:06:45.829 INFO:tasks.workunit.client.0.smithi157.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T12:06:45.835 INFO:tasks.workunit.client.0.smithi157.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T12:06:45.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T12:06:45.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T12:06:45.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T12:06:45.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T12:06:45.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T12:06:45.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T12:06:45.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T12:06:45.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T12:06:45.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/ver-health 2024-09-18T12:06:45.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:20: run: local dir=td/ver-health 2024-09-18T12:06:45.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:21: run: shift 2024-09-18T12:06:45.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: export CEPH_MON_A=127.0.0.1:7165 2024-09-18T12:06:45.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: CEPH_MON_A=127.0.0.1:7165 2024-09-18T12:06:45.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: export CEPH_MON_B=127.0.0.1:7166 2024-09-18T12:06:45.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: CEPH_MON_B=127.0.0.1:7166 2024-09-18T12:06:45.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:25: run: export CEPH_ARGS 2024-09-18T12:06:45.837 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: uuidgen 2024-09-18T12:06:45.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: CEPH_ARGS+='--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none ' 2024-09-18T12:06:45.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:27: run: CEPH_ARGS+='--mon-host= ' 2024-09-18T12:06:45.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:28: run: CEPH_ARGS+='--mon_health_to_clog_tick_interval=1.0 ' 2024-09-18T12:06:45.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: export 'ORIG_CEPH_ARGS=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2024-09-18T12:06:45.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: ORIG_CEPH_ARGS='--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2024-09-18T12:06:45.838 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: set 2024-09-18T12:06:45.839 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T12:06:45.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: local 'funcs=TEST_check_version_health_1 2024-09-18T12:06:45.839 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_check_version_health_2 2024-09-18T12:06:45.839 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_check_version_health_3' 2024-09-18T12:06:45.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-09-18T12:06:45.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-09-18T12:06:45.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-18T12:06:45.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-18T12:06:45.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-18T12:06:45.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:06:45.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-18T12:06:45.839 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:06:45.840 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:06:45.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:06:45.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:06:45.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:06:45.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:06:45.841 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:06:45.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:06:45.841 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:06:45.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:06:45.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:06:45.842 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:06:45.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:06:45.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:06:45.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:06:45.842 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:06:45.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:06:45.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:06:45.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-18T12:06:45.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:06:45.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:06:45.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:06:45.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188934 2024-09-18T12:06:45.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:06:45.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:06:45.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-18T12:06:45.847 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T12:06:45.847 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:06:45.847 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:06:45.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.188934 2024-09-18T12:06:45.850 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T12:06:45.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T12:06:45.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T12:06:45.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T12:06:45.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-18T12:06:45.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_1 td/ver-health 2024-09-18T12:06:45.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:68: TEST_check_version_health_1: local dir=td/ver-health 2024-09-18T12:06:45.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:71: TEST_check_version_health_1: CEPH_ARGS='--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:06:45.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:73: TEST_check_version_health_1: setup td/ver-health 2024-09-18T12:06:45.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-18T12:06:45.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-18T12:06:45.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-18T12:06:45.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:06:45.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-18T12:06:45.851 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:06:45.851 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:06:45.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:06:45.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:06:45.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:06:45.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:06:45.852 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:06:45.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:06:45.852 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:06:45.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:06:45.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:06:45.853 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:06:45.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:06:45.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:06:45.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:06:45.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:06:45.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:06:45.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:06:45.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-18T12:06:45.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:06:45.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:06:45.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:06:45.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188934 2024-09-18T12:06:45.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:06:45.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:06:45.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-18T12:06:45.859 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T12:06:45.859 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:06:45.859 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:06:45.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.188934 2024-09-18T12:06:45.861 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T12:06:45.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T12:06:45.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T12:06:45.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-18T12:06:45.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:76: TEST_check_version_health_1: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-18T12:06:45.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-18T12:06:45.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T12:06:45.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T12:06:45.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T12:06:45.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-09-18T12:06:45.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-18T12:06:46.229 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T12:06:46.230 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:06:46.230 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:06:46.230 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:06:46.230 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:06:46.230 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:06:46.230 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:06:46.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-18T12:06:46.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T12:06:46.491 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T12:06:46.491 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:06:46.491 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T12:06:46.491 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T12:06:46.491 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T12:06:46.491 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T12:06:46.491 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T12:06:46.491 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T12:06:46.492 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:06:46.492 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:06:46.492 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:06:46.492 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188934/ceph-mon.a.asok 2024-09-18T12:06:46.492 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:06:46.492 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188934/ceph-mon.a.asok config get fsid 2024-09-18T12:06:46.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T12:06:46.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:06:46.567 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T12:06:46.567 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T12:06:46.567 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T12:06:46.567 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T12:06:46.567 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T12:06:46.567 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T12:06:46.567 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:06:46.567 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:06:46.567 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:06:46.567 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188934/ceph-mon.a.asok 2024-09-18T12:06:46.567 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:06:46.567 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188934/ceph-mon.a.asok config get mon_host 2024-09-18T12:06:46.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:77: TEST_check_version_health_1: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-18T12:06:46.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-18T12:06:46.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T12:06:46.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-18T12:06:46.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T12:06:46.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-09-18T12:06:46.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-18T12:06:47.294 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T12:06:47.294 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:06:47.294 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:06:47.294 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:06:47.294 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:06:47.294 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:06:47.294 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:06:47.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-18T12:06:47.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T12:06:47.697 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-18T12:06:47.697 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:06:47.697 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-18T12:06:47.698 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T12:06:47.699 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T12:06:47.699 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-18T12:06:47.699 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-18T12:06:47.699 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-18T12:06:47.699 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:06:47.699 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:06:47.699 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:06:47.699 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188934/ceph-mon.b.asok 2024-09-18T12:06:47.699 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:06:47.699 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188934/ceph-mon.b.asok config get fsid 2024-09-18T12:06:47.777 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-18T12:06:47.777 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:06:47.777 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-18T12:06:47.778 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T12:06:47.778 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T12:06:47.778 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-18T12:06:47.778 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-18T12:06:47.778 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-18T12:06:47.778 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:06:47.778 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:06:47.778 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:06:47.778 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188934/ceph-mon.b.asok 2024-09-18T12:06:47.778 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:06:47.778 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188934/ceph-mon.b.asok config get mon_host 2024-09-18T12:06:47.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:78: TEST_check_version_health_1: run_osd td/ver-health 0 2024-09-18T12:06:47.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-18T12:06:47.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:06:47.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T12:06:47.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:06:47.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-09-18T12:06:47.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:06:47.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:06:47.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:06:47.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:06:47.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-18T12:06:47.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-18T12:06:47.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:06:47.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T12:06:47.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:06:47.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:06:47.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:06:47.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:06:47.855 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:06:47.855 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:06:47.855 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:06:47.855 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:06:47.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:06:47.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:06:47.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:06:47.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:06:47.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:06:47.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:06:47.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:06:47.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:06:47.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:06:47.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:06:47.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:06:47.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T12:06:47.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-09-18T12:06:47.856 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:06:47.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=543734c1-f7a1-4721-b82e-ede1ae728b23 2024-09-18T12:06:47.856 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 543734c1-f7a1-4721-b82e-ede1ae728b23 2024-09-18T12:06:47.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 543734c1-f7a1-4721-b82e-ede1ae728b23' 2024-09-18T12:06:47.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:06:47.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBXwupm0c6uMxAAOck+3t3ZhZl/O+QEcAVuLw== 2024-09-18T12:06:47.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBXwupm0c6uMxAAOck+3t3ZhZl/O+QEcAVuLw=="}' 2024-09-18T12:06:47.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 543734c1-f7a1-4721-b82e-ede1ae728b23 -i td/ver-health/0/new.json 2024-09-18T12:06:54.177 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:06:54.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-09-18T12:06:54.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXwupm0c6uMxAAOck+3t3ZhZl/O+QEcAVuLw== --osd-uuid 543734c1-f7a1-4721-b82e-ede1ae728b23 2024-09-18T12:06:54.212 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:06:54.210+0000 7f0336521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:06:54.213 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:06:54.212+0000 7f0336521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:06:54.214 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:06:54.213+0000 7f0336521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:06:54.215 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:06:54.213+0000 7f0336521440 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-18T12:06:54.215 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:06:54.213+0000 7f0336521440 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-18T12:06:54.215 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:06:54.213+0000 7f0336521440 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-18T12:06:54.315 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:06:54.314+0000 7f0336521440 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2024-09-18T12:06:59.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-09-18T12:06:59.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:06:59.994 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T12:06:59.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T12:06:59.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:07:00.188 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T12:07:00.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T12:07:00.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:07:00.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:07:00.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:07:00.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:07:00.210 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:00.209+0000 7f80b900b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:07:00.217 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:00.216+0000 7f80b900b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:07:00.218 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:00.217+0000 7f80b900b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:07:00.371 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T12:07:00.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:07:00.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T12:07:00.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:07:00.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:07:00.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:00.372 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:07:00.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:07:00.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:00.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:00.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:00.748 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:00.747+0000 7f80b900b440 -1 Falling back to public interface 2024-09-18T12:07:01.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:01.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:01.599 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:07:01.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:07:01.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:01.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:01.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:02.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:02.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:02.779 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:07:02.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:07:02.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:02.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:02.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:03.374 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:03.373+0000 7f80b900b440 -1 osd.0 0 log_to_monitors true 2024-09-18T12:07:03.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:03.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:03.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:07:03.963 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:07:03.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:03.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:04.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:05.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:05.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:05.271 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T12:07:05.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T12:07:05.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:05.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:05.458 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:06.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:06.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:06.460 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T12:07:06.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T12:07:06.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:06.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:06.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:07.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:07.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:07.646 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T12:07:07.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T12:07:07.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:07.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:07.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:08.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:08.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:08.833 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T12:07:08.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T12:07:08.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:08.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:09.019 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:10.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:10.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:10.021 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T12:07:10.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T12:07:10.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:10.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:10.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:11.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:11.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:11.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T12:07:11.209 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T12:07:11.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:11.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:11.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:12.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:12.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:12.397 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T12:07:12.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T12:07:12.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:12.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:12.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:13.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:13.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:13.586 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T12:07:13.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T12:07:13.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:13.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:13.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:14.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:14.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:14.775 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T12:07:14.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T12:07:14.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:14.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:14.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:15.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:15.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:15.962 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T12:07:15.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T12:07:15.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:15.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:16.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:17.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:17.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:17.143 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T12:07:17.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T12:07:17.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:17.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:17.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:18.325 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:18.325 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:18.325 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T12:07:18.325 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T12:07:18.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:18.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:18.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:19.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:19.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:19.510 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T12:07:19.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T12:07:19.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:19.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:19.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:20.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:20.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:20.701 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T12:07:20.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T12:07:20.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:20.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:20.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:21.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:21.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:21.887 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T12:07:21.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T12:07:21.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:21.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:22.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:23.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:23.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:23.076 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-18T12:07:23.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T12:07:23.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:23.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:23.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:24.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:24.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:24.405 INFO:tasks.workunit.client.0.smithi157.stdout:20 2024-09-18T12:07:24.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T12:07:24.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:24.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:24.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:25.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:25.594 INFO:tasks.workunit.client.0.smithi157.stdout:21 2024-09-18T12:07:25.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:25.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-18T12:07:25.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:25.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:25.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:26.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:26.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:26.808 INFO:tasks.workunit.client.0.smithi157.stdout:22 2024-09-18T12:07:26.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-18T12:07:26.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:26.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:07:26.992 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/189972,v1:127.0.0.1:6801/189972] [v2:127.0.0.1:6802/189972,v1:127.0.0.1:6803/189972] exists,up 543734c1-f7a1-4721-b82e-ede1ae728b23 2024-09-18T12:07:26.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:07:26.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:07:26.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:07:26.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:79: TEST_check_version_health_1: run_osd td/ver-health 1 2024-09-18T12:07:26.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-18T12:07:26.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:07:26.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T12:07:26.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:07:26.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-09-18T12:07:26.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:07:26.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:07:26.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:07:26.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:07:26.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-18T12:07:26.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-18T12:07:26.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:07:26.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T12:07:26.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:07:26.998 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:07:26.998 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:07:26.998 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:07:26.998 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:07:26.998 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:07:26.998 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:07:26.998 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:07:26.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:07:26.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:07:26.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:07:26.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:07:26.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:07:26.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:07:26.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:07:26.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:07:26.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:07:26.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:07:26.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:07:26.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T12:07:26.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-09-18T12:07:26.999 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:07:26.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a713b246-c0f1-4713-9c01-9e9ee23b4e86 2024-09-18T12:07:26.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a713b246-c0f1-4713-9c01-9e9ee23b4e86' 2024-09-18T12:07:27.000 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 a713b246-c0f1-4713-9c01-9e9ee23b4e86 2024-09-18T12:07:27.000 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:07:27.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB/wupmn7h2ABAA2eq5NiDcuHA6aZjbtU7zlQ== 2024-09-18T12:07:27.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB/wupmn7h2ABAA2eq5NiDcuHA6aZjbtU7zlQ=="}' 2024-09-18T12:07:27.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a713b246-c0f1-4713-9c01-9e9ee23b4e86 -i td/ver-health/1/new.json 2024-09-18T12:07:27.338 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:07:27.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-09-18T12:07:27.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB/wupmn7h2ABAA2eq5NiDcuHA6aZjbtU7zlQ== --osd-uuid a713b246-c0f1-4713-9c01-9e9ee23b4e86 2024-09-18T12:07:27.371 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:27.370+0000 7f0975208440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:07:27.373 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:27.372+0000 7f0975208440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:07:27.374 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:27.372+0000 7f0975208440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:07:27.374 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:27.373+0000 7f0975208440 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-18T12:07:27.374 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:27.373+0000 7f0975208440 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-18T12:07:27.374 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:27.373+0000 7f0975208440 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-18T12:07:27.460 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:27.459+0000 7f0975208440 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2024-09-18T12:07:33.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-09-18T12:07:33.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:07:33.316 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T12:07:33.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T12:07:33.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:07:33.512 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T12:07:33.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T12:07:33.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:07:33.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:07:33.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:07:33.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:07:33.534 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:33.532+0000 7f1b6ca12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:07:33.541 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:33.540+0000 7f1b6ca12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:07:33.542 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:33.541+0000 7f1b6ca12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:07:33.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T12:07:33.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:07:33.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T12:07:33.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:07:33.743 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:07:33.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:07:33.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:33.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:07:33.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:33.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:33.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:34.074 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:34.073+0000 7f1b6ca12440 -1 Falling back to public interface 2024-09-18T12:07:34.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:34.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:34.980 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:07:34.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:07:34.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:34.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:35.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:36.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:36.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:36.170 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:07:36.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:07:36.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:36.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:36.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:36.689 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:07:36.688+0000 7f1b6ca12440 -1 osd.1 0 log_to_monitors true 2024-09-18T12:07:37.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:37.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:37.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:07:37.357 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:07:37.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:37.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:37.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:38.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:38.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:38.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T12:07:38.539 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T12:07:38.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:38.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:38.724 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:39.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:39.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:39.729 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T12:07:39.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T12:07:39.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:39.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:39.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:40.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:40.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:40.912 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T12:07:40.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T12:07:40.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:40.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:41.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:42.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:42.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:42.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T12:07:42.100 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T12:07:42.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:42.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:42.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:43.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:43.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:43.285 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T12:07:43.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T12:07:43.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:43.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:43.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:44.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:44.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:44.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T12:07:44.480 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T12:07:44.480 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:44.480 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:44.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:45.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:45.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:45.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T12:07:45.667 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T12:07:45.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:45.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:45.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:46.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:46.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:46.851 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T12:07:46.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T12:07:46.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:46.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:47.030 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:48.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:48.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:48.032 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T12:07:48.033 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T12:07:48.033 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:48.033 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:48.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:49.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T12:07:49.219 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T12:07:49.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:49.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:49.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:50.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:50.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:50.402 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T12:07:50.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T12:07:50.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:50.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:50.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:51.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:51.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:51.588 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T12:07:51.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T12:07:51.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:51.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:51.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:52.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:52.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:52.769 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T12:07:52.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T12:07:52.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:52.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:52.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:53.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:53.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:53.958 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T12:07:53.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T12:07:53.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:53.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:54.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:55.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:55.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:55.145 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T12:07:55.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T12:07:55.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:55.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:55.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:56.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:56.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:56.330 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-18T12:07:56.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T12:07:56.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:56.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:56.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:57.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:57.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:57.514 INFO:tasks.workunit.client.0.smithi157.stdout:20 2024-09-18T12:07:57.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T12:07:57.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:57.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:57.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:58.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:58.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:58.701 INFO:tasks.workunit.client.0.smithi157.stdout:21 2024-09-18T12:07:58.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-18T12:07:58.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:58.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:07:58.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:07:59.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:07:59.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:07:59.889 INFO:tasks.workunit.client.0.smithi157.stdout:22 2024-09-18T12:07:59.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-18T12:07:59.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:07:59.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:08:00.146 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/191853,v1:127.0.0.1:6809/191853] [v2:127.0.0.1:6810/191853,v1:127.0.0.1:6811/191853] exists,up a713b246-c0f1-4713-9c01-9e9ee23b4e86 2024-09-18T12:08:00.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:08:00.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:08:00.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:08:00.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:80: TEST_check_version_health_1: run_osd td/ver-health 2 2024-09-18T12:08:00.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-18T12:08:00.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:08:00.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T12:08:00.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:08:00.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-09-18T12:08:00.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:08:00.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:08:00.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:08:00.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:08:00.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-18T12:08:00.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-18T12:08:00.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:08:00.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T12:08:00.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:08:00.155 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:08:00.155 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:08:00.155 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:08:00.155 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:08:00.155 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:08:00.155 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:08:00.155 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:08:00.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:08:00.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:08:00.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:08:00.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:08:00.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:08:00.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:08:00.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:08:00.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:08:00.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:08:00.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:08:00.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:08:00.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T12:08:00.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-09-18T12:08:00.156 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:08:00.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2a72e7d1-338f-4487-b861-b7b3dc6d6baf 2024-09-18T12:08:00.156 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 2a72e7d1-338f-4487-b861-b7b3dc6d6baf 2024-09-18T12:08:00.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2a72e7d1-338f-4487-b861-b7b3dc6d6baf' 2024-09-18T12:08:00.157 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:08:00.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCgwupmh6e4CRAAMxCKrs/6imLQXdP4uBaA4A== 2024-09-18T12:08:00.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCgwupmh6e4CRAAMxCKrs/6imLQXdP4uBaA4A=="}' 2024-09-18T12:08:00.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2a72e7d1-338f-4487-b861-b7b3dc6d6baf -i td/ver-health/2/new.json 2024-09-18T12:08:00.856 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:08:00.865 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-09-18T12:08:00.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCgwupmh6e4CRAAMxCKrs/6imLQXdP4uBaA4A== --osd-uuid 2a72e7d1-338f-4487-b861-b7b3dc6d6baf 2024-09-18T12:08:00.889 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:00.888+0000 7f317720a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:08:00.990 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:00.989+0000 7f317720a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:08:00.991 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:00.990+0000 7f317720a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:08:00.992 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:00.990+0000 7f317720a440 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-18T12:08:00.992 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:00.990+0000 7f317720a440 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-18T12:08:00.992 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:00.990+0000 7f317720a440 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-18T12:08:01.082 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:01.080+0000 7f317720a440 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2024-09-18T12:08:06.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-09-18T12:08:06.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:08:06.819 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-18T12:08:06.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T12:08:06.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:08:07.014 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-18T12:08:07.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T12:08:07.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:08:07.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:08:07.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:08:07.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:08:07.036 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:07.034+0000 7f6f1ec12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:08:07.045 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:07.044+0000 7f6f1ec12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:08:07.046 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:07.045+0000 7f6f1ec12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:08:07.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T12:08:07.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:08:07.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T12:08:07.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:08:07.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:08:07.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:07.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:08:07.209 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:08:07.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:07.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:07.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:07.578 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:07.577+0000 7f6f1ec12440 -1 Falling back to public interface 2024-09-18T12:08:08.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:08.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:08.443 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:08:08.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:08:08.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:08.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:08.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:09.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:09.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:09.623 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:08:09.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:08:09.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:09.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:09.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:10.109 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:10.107+0000 7f6f1ec12440 -1 osd.2 0 log_to_monitors true 2024-09-18T12:08:10.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:10.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:10.799 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:08:10.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:08:10.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:10.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:11.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:12.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:12.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:12.104 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T12:08:12.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T12:08:12.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:12.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:12.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:13.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:13.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:13.290 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T12:08:13.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T12:08:13.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:13.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:13.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:14.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:14.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:14.476 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T12:08:14.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T12:08:14.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:14.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:14.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:15.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:15.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:15.663 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T12:08:15.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T12:08:15.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:15.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:15.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:16.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:16.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:16.852 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T12:08:16.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T12:08:16.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:16.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:17.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:18.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:18.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:18.041 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T12:08:18.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T12:08:18.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:18.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:18.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:19.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:19.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:19.230 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T12:08:19.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T12:08:19.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:19.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:19.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:20.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:20.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:20.416 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T12:08:20.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T12:08:20.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:20.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:20.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:21.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:21.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:21.607 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T12:08:21.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T12:08:21.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:21.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:21.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:22.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:22.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:22.797 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T12:08:22.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T12:08:22.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:22.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:22.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:23.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:23.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:23.986 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T12:08:23.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T12:08:23.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:23.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:24.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:25.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:25.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:25.301 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T12:08:25.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T12:08:25.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:25.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:25.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:26.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:26.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:26.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T12:08:26.487 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T12:08:26.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:26.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:26.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:27.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:27.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:27.674 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T12:08:27.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T12:08:27.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:27.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:27.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:28.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:28.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:28.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T12:08:28.861 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T12:08:28.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:28.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:29.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:30.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:30.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:30.047 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-18T12:08:30.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T12:08:30.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:30.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:30.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:31.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:31.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:31.237 INFO:tasks.workunit.client.0.smithi157.stdout:20 2024-09-18T12:08:31.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T12:08:31.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:31.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:31.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:32.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:32.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:32.423 INFO:tasks.workunit.client.0.smithi157.stdout:21 2024-09-18T12:08:32.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-18T12:08:32.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:32.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:32.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:33.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:33.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:33.608 INFO:tasks.workunit.client.0.smithi157.stdout:22 2024-09-18T12:08:33.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-18T12:08:33.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:33.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:33.802 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/193732,v1:127.0.0.1:6817/193732] [v2:127.0.0.1:6818/193732,v1:127.0.0.1:6819/193732] exists,up 2a72e7d1-338f-4487-b861-b7b3dc6d6baf 2024-09-18T12:08:33.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:08:33.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:08:33.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:08:33.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:82: TEST_check_version_health_1: sleep 5 2024-09-18T12:08:38.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:83: TEST_check_version_health_1: ceph health detail 2024-09-18T12:08:38.986 INFO:tasks.workunit.client.0.smithi157.stdout:HEALTH_OK 2024-09-18T12:08:38.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: ceph health detail 2024-09-18T12:08:38.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: grep DAEMON_OLD_VERSION 2024-09-18T12:08:39.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:87: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.1 2024-09-18T12:08:39.226 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:08:39.226 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:08:39.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:08:39.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:08:39.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:08:39.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:08:39.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-18T12:08:39.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: activate_osd td/ver-health 1 2024-09-18T12:08:39.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-18T12:08:39.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T12:08:39.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T12:08:39.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T12:08:39.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/1 2024-09-18T12:08:39.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:08:39.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:08:39.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:08:39.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:08:39.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-18T12:08:39.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-18T12:08:39.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T12:08:39.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T12:08:39.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:08:39.335 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T12:08:39.335 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:08:39.335 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:08:39.335 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:08:39.335 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:08:39.335 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:08:39.335 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:08:39.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:08:39.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:08:39.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:08:39.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:08:39.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:08:39.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:08:39.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:08:39.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:08:39.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T12:08:39.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T12:08:39.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/1 2024-09-18T12:08:39.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T12:08:39.337 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T12:08:39.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:08:39.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/1/whoami 2024-09-18T12:08:39.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T12:08:39.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T12:08:39.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T12:08:39.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T12:08:39.355 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:39.354+0000 7fdcc5764440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:08:39.401 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:39.400+0000 7fdcc5764440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:08:39.402 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:39.401+0000 7fdcc5764440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:08:39.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T12:08:39.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:08:39.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T12:08:39.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:08:39.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:08:39.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:39.571 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:08:39.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:08:39.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:39.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:08:39.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:39.916 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:39.915+0000 7fdcc5764440 -1 Falling back to public interface 2024-09-18T12:08:40.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:40.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:40.814 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:08:40.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:08:40.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:40.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:08:40.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:41.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:41.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:41.994 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:08:41.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:08:41.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:41.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:08:42.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:42.887 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:42.885+0000 7fdcc5764440 -1 osd.1 16 log_to_monitors true 2024-09-18T12:08:43.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:43.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:43.169 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:08:43.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:08:43.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:43.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:08:43.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:44.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:44.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:44.398 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T12:08:44.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T12:08:44.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:44.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:08:44.575 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [10,16) [v2:127.0.0.1:6808/194844,v1:127.0.0.1:6809/194844] [v2:127.0.0.1:6810/194844,v1:127.0.0.1:6811/194844] exists,up a713b246-c0f1-4713-9c01-9e9ee23b4e86 2024-09-18T12:08:44.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:08:44.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:08:44.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:08:44.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:90: TEST_check_version_health_1: wait_for_health_string 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:08:44.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:08:44.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-18T12:08:44.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-18T12:08:44.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-18T12:08:44.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-18T12:08:44.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:08:44.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:08:44.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:08:44.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:08:45.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:08:45.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:08:45.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:08:45.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:08:45.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:08:46.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:08:46.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:08:46.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:08:46.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:08:47.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:08:48.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:08:48.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:08:48.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:08:48.218 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:08:48.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:08:49.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:08:49.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:08:49.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:08:49.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:08:49.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:08:50.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:08:50.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:08:50.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:08:50.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:08:50.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:08:51.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:08:51.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:08:51.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:08:51.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:08:52.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-18T12:08:52.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-18T12:08:52.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-18T12:08:52.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-18T12:08:52.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-18T12:08:52.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:92: TEST_check_version_health_1: ceph health detail 2024-09-18T12:08:52.220 INFO:tasks.workunit.client.0.smithi157.stdout:HEALTH_WARN There is a daemon running an older version of ceph; no active mgr 2024-09-18T12:08:52.220 INFO:tasks.workunit.client.0.smithi157.stdout:[WRN] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph 2024-09-18T12:08:52.220 INFO:tasks.workunit.client.0.smithi157.stdout: osd.1 is running an older version of ceph: 01.00.00-gversion-test 2024-09-18T12:08:52.220 INFO:tasks.workunit.client.0.smithi157.stdout:[WRN] MGR_DOWN: no active mgr 2024-09-18T12:08:52.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: ceph health detail 2024-09-18T12:08:52.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:08:52.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: ceph health detail 2024-09-18T12:08:52.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph' 2024-09-18T12:08:52.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: ceph health detail 2024-09-18T12:08:52.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: grep -q 'osd.1 is running an older version of ceph: 01.00.00-gversion-test' 2024-09-18T12:08:53.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:98: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.2 2024-09-18T12:08:53.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:08:53.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:08:53.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:08:53.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:08:53.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:08:53.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:08:53.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-18T12:08:53.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: activate_osd td/ver-health 2 2024-09-18T12:08:53.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-18T12:08:53.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T12:08:53.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T12:08:53.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T12:08:53.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2024-09-18T12:08:53.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:08:53.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:08:53.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:08:53.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:08:53.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-18T12:08:53.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-18T12:08:53.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T12:08:53.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T12:08:53.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:08:53.127 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T12:08:53.127 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:08:53.127 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:08:53.127 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:08:53.127 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:08:53.127 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:08:53.127 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:08:53.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:08:53.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:08:53.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:08:53.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:08:53.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:08:53.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:08:53.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:08:53.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:08:53.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T12:08:53.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T12:08:53.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2024-09-18T12:08:53.128 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-18T12:08:53.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T12:08:53.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:08:53.129 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2024-09-18T12:08:53.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T12:08:53.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T12:08:53.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T12:08:53.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T12:08:53.146 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:53.145+0000 7f2e3880e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:08:53.232 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:53.230+0000 7f2e3880e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:08:53.233 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:53.231+0000 7f2e3880e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:08:53.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T12:08:53.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:08:53.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T12:08:53.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:08:53.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:08:53.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:53.401 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:08:53.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:08:53.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:53.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:53.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:53.747 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:53.746+0000 7f2e3880e440 -1 Falling back to public interface 2024-09-18T12:08:54.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:54.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:54.578 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:08:54.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:08:54.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:54.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:54.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:55.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:55.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:55.765 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:08:55.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:08:55.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:55.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:55.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:56.617 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:56.616+0000 7f2e3880e440 -1 osd.2 19 log_to_monitors true 2024-09-18T12:08:56.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:56.944 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:08:56.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:56.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:08:56.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:56.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:08:57.211 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 20 last_clean_interval [15,19) [v2:127.0.0.1:6816/195741,v1:127.0.0.1:6817/195741] [v2:127.0.0.1:6818/195741,v1:127.0.0.1:6819/195741] exists,up 2a72e7d1-338f-4487-b861-b7b3dc6d6baf 2024-09-18T12:08:57.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:08:57.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:08:57.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:08:57.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:100: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.0 2024-09-18T12:08:57.213 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:08:57.213 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:08:57.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:08:57.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:08:57.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:08:57.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:08:57.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: ceph_debug_version_for_testing=02.00.00-gversion-test 2024-09-18T12:08:57.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: activate_osd td/ver-health 0 2024-09-18T12:08:57.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-18T12:08:57.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T12:08:57.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T12:08:57.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T12:08:57.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2024-09-18T12:08:57.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:08:57.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:08:57.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:08:57.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:08:57.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-18T12:08:57.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-18T12:08:57.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T12:08:57.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T12:08:57.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:08:57.321 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T12:08:57.322 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:08:57.322 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:08:57.322 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:08:57.322 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:08:57.322 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:08:57.322 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:08:57.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:08:57.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:08:57.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:08:57.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:08:57.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:08:57.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:08:57.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:08:57.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:08:57.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T12:08:57.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T12:08:57.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2024-09-18T12:08:57.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T12:08:57.323 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T12:08:57.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:08:57.324 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2024-09-18T12:08:57.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T12:08:57.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T12:08:57.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T12:08:57.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T12:08:57.342 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:57.340+0000 7fe03bd21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:08:57.344 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:57.342+0000 7fe03bd21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:08:57.345 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:57.344+0000 7fe03bd21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:08:57.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T12:08:57.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:08:57.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T12:08:57.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:08:57.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:08:57.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:57.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:08:57.627 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:08:57.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:57.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:08:57.869 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:08:57.868+0000 7fe03bd21440 -1 Falling back to public interface 2024-09-18T12:08:57.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:08:58.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:08:58.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:08:58.996 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:08:58.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:08:58.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:08:58.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:08:59.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:00.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:00.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:00.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:09:00.175 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:09:00.175 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:00.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:00.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:00.817 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:00.815+0000 7fe03bd21440 -1 osd.0 21 log_to_monitors true 2024-09-18T12:09:01.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:01.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:01.353 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:09:01.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:09:01.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:01.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:01.649 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 23 last_clean_interval [5,21) [v2:127.0.0.1:6800/196311,v1:127.0.0.1:6801/196311] [v2:127.0.0.1:6802/196311,v1:127.0.0.1:6803/196311] exists,up 543734c1-f7a1-4721-b82e-ede1ae728b23 2024-09-18T12:09:01.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:09:01.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:09:01.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:09:01.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:103: TEST_check_version_health_1: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-18T12:09:01.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-18T12:09:01.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-18T12:09:01.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-18T12:09:01.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-18T12:09:01.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-18T12:09:01.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:09:01.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:09:01.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-18T12:09:01.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:09:02.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:09:02.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:09:02.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:09:02.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-18T12:09:03.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-18T12:09:03.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-18T12:09:03.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-18T12:09:03.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-18T12:09:03.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-18T12:09:03.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:105: TEST_check_version_health_1: ceph health detail 2024-09-18T12:09:03.238 INFO:tasks.workunit.client.0.smithi157.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph; no active mgr 2024-09-18T12:09:03.239 INFO:tasks.workunit.client.0.smithi157.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2024-09-18T12:09:03.239 INFO:tasks.workunit.client.0.smithi157.stdout: osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test 2024-09-18T12:09:03.239 INFO:tasks.workunit.client.0.smithi157.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2024-09-18T12:09:03.239 INFO:tasks.workunit.client.0.smithi157.stdout:[WRN] MGR_DOWN: no active mgr 2024-09-18T12:09:03.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: ceph health detail 2024-09-18T12:09:03.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-18T12:09:03.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: ceph health detail 2024-09-18T12:09:03.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2024-09-18T12:09:03.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: ceph health detail 2024-09-18T12:09:03.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: grep -q 'osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test' 2024-09-18T12:09:04.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: ceph health detail 2024-09-18T12:09:04.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2024-09-18T12:09:04.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2024-09-18T12:09:04.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-18T12:09:04.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:09:04.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-18T12:09:04.262 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:09:04.262 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:09:04.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:09:04.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:09:04.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:09:04.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:09:04.369 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:09:04.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:09:04.370 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:09:04.371 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:09:04.371 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:09:04.372 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:09:04.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:09:04.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:09:04.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:09:04.373 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:09:04.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:09:04.375 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:09:04.375 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-18T12:09:04.390 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:09:04.390 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:09:04.390 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:09:04.390 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188934 2024-09-18T12:09:04.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:09:04.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:09:04.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-09-18T12:09:04.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-09-18T12:09:04.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-18T12:09:04.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-18T12:09:04.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-18T12:09:04.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:09:04.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-18T12:09:04.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:09:04.393 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:09:04.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:09:04.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:09:04.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:09:04.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:09:04.394 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:09:04.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:09:04.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:09:04.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:09:04.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:09:04.396 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:09:04.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:09:04.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:09:04.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:09:04.397 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:09:04.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:09:04.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:09:04.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-18T12:09:04.400 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:09:04.400 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:09:04.400 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:09:04.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188934 2024-09-18T12:09:04.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:09:04.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:09:04.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-18T12:09:04.403 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T12:09:04.403 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:09:04.403 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:09:04.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.188934 2024-09-18T12:09:04.405 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T12:09:04.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T12:09:04.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T12:09:04.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-18T12:09:04.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_2 td/ver-health 2024-09-18T12:09:04.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:114: TEST_check_version_health_2: local dir=td/ver-health 2024-09-18T12:09:04.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:117: TEST_check_version_health_2: CEPH_ARGS='--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:09:04.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:119: TEST_check_version_health_2: setup td/ver-health 2024-09-18T12:09:04.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-18T12:09:04.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-18T12:09:04.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-18T12:09:04.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:09:04.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-18T12:09:04.407 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:09:04.407 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:09:04.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:09:04.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:09:04.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:09:04.408 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:09:04.408 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:09:04.408 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:09:04.408 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:09:04.408 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:09:04.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:09:04.409 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:09:04.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:09:04.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:09:04.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:09:04.410 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:09:04.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:09:04.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:09:04.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-18T12:09:04.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:09:04.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:09:04.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:09:04.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188934 2024-09-18T12:09:04.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:09:04.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:09:04.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-18T12:09:04.415 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T12:09:04.415 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:09:04.415 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:09:04.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.188934 2024-09-18T12:09:04.417 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T12:09:04.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T12:09:04.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T12:09:04.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-18T12:09:04.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:122: TEST_check_version_health_2: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-18T12:09:04.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-18T12:09:04.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T12:09:04.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T12:09:04.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T12:09:04.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-09-18T12:09:04.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-18T12:09:04.773 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T12:09:04.773 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:09:04.773 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:09:04.773 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:09:04.773 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:09:04.773 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:09:04.774 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:09:04.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-18T12:09:05.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T12:09:05.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T12:09:05.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:09:05.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T12:09:05.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T12:09:05.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T12:09:05.051 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T12:09:05.051 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T12:09:05.051 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T12:09:05.051 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:09:05.051 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:09:05.051 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:09:05.052 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188934/ceph-mon.a.asok 2024-09-18T12:09:05.052 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:09:05.052 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188934/ceph-mon.a.asok config get fsid 2024-09-18T12:09:05.124 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T12:09:05.124 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:09:05.124 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T12:09:05.124 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T12:09:05.124 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T12:09:05.125 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T12:09:05.125 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T12:09:05.125 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T12:09:05.125 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:09:05.125 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:09:05.126 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:09:05.126 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188934/ceph-mon.a.asok 2024-09-18T12:09:05.126 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:09:05.126 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188934/ceph-mon.a.asok config get mon_host 2024-09-18T12:09:05.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:123: TEST_check_version_health_2: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-18T12:09:05.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-18T12:09:05.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T12:09:05.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-18T12:09:05.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T12:09:05.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-09-18T12:09:05.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-18T12:09:05.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T12:09:05.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:09:05.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:09:05.854 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:09:05.854 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:09:05.854 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:09:05.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:09:05.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-18T12:09:06.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T12:09:06.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-18T12:09:06.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:09:06.284 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-18T12:09:06.284 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T12:09:06.285 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T12:09:06.285 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-18T12:09:06.285 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-18T12:09:06.285 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-18T12:09:06.285 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:09:06.285 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:09:06.285 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:09:06.285 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188934/ceph-mon.b.asok 2024-09-18T12:09:06.286 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:09:06.286 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188934/ceph-mon.b.asok config get fsid 2024-09-18T12:09:06.369 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-18T12:09:06.369 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:09:06.369 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-18T12:09:06.369 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T12:09:06.369 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T12:09:06.369 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-18T12:09:06.369 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-18T12:09:06.369 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-18T12:09:06.369 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:09:06.369 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:09:06.370 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:09:06.370 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188934/ceph-mon.b.asok 2024-09-18T12:09:06.370 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:09:06.370 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188934/ceph-mon.b.asok config get mon_host 2024-09-18T12:09:06.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:124: TEST_check_version_health_2: run_osd td/ver-health 0 2024-09-18T12:09:06.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-18T12:09:06.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:09:06.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T12:09:06.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:09:06.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-09-18T12:09:06.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:09:06.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:09:06.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:09:06.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:09:06.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-18T12:09:06.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-18T12:09:06.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:09:06.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T12:09:06.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:09:06.450 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:09:06.450 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:09:06.450 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:09:06.450 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:09:06.450 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:09:06.450 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:09:06.450 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:09:06.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:09:06.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:09:06.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:09:06.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:09:06.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:09:06.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:09:06.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:09:06.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:09:06.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:09:06.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:09:06.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:09:06.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T12:09:06.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-09-18T12:09:06.451 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:09:06.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=65936d02-f832-40b7-9b7a-c5f99ff973ef 2024-09-18T12:09:06.452 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 65936d02-f832-40b7-9b7a-c5f99ff973ef 2024-09-18T12:09:06.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 65936d02-f832-40b7-9b7a-c5f99ff973ef' 2024-09-18T12:09:06.452 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:09:06.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDiwupmxjh7GxAA/Hfi0u3ySsesK2UZox8P8Q== 2024-09-18T12:09:06.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDiwupmxjh7GxAA/Hfi0u3ySsesK2UZox8P8Q=="}' 2024-09-18T12:09:06.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 65936d02-f832-40b7-9b7a-c5f99ff973ef -i td/ver-health/0/new.json 2024-09-18T12:09:12.764 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:09:12.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-09-18T12:09:12.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDiwupmxjh7GxAA/Hfi0u3ySsesK2UZox8P8Q== --osd-uuid 65936d02-f832-40b7-9b7a-c5f99ff973ef 2024-09-18T12:09:12.798 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:12.797+0000 7f37d261c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:09:12.800 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:12.799+0000 7f37d261c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:09:12.801 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:12.800+0000 7f37d261c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:09:12.802 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:12.800+0000 7f37d261c440 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-18T12:09:12.802 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:12.800+0000 7f37d261c440 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-18T12:09:12.802 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:12.800+0000 7f37d261c440 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-18T12:09:12.889 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:12.888+0000 7f37d261c440 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2024-09-18T12:09:18.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-09-18T12:09:18.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:09:18.497 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T12:09:18.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T12:09:18.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:09:18.690 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T12:09:18.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T12:09:18.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:09:18.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:09:18.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:09:18.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:09:18.712 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:18.711+0000 7f6e2b54c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:09:18.726 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:18.724+0000 7f6e2b54c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:09:18.727 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:18.726+0000 7f6e2b54c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:09:18.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T12:09:18.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:09:18.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T12:09:18.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:09:18.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:09:18.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:18.871 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:09:18.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:09:18.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:18.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:19.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:19.252 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:19.251+0000 7f6e2b54c440 -1 Falling back to public interface 2024-09-18T12:09:20.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:20.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:20.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:09:20.097 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:09:20.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:20.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:20.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:21.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:21.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:21.285 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:09:21.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:09:21.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:21.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:21.463 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:21.881 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:21.879+0000 7f6e2b54c440 -1 osd.0 0 log_to_monitors true 2024-09-18T12:09:22.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:22.465 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:09:22.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:22.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:09:22.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:22.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:22.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:23.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:23.694 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T12:09:23.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:23.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T12:09:23.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:23.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:23.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:24.881 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:24.881 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:24.881 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T12:09:24.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T12:09:24.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:24.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:25.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:26.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:26.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:26.068 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T12:09:26.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T12:09:26.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:26.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:26.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:27.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:27.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:27.257 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T12:09:27.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T12:09:27.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:27.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:27.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:28.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:28.444 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T12:09:28.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:28.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T12:09:28.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:28.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:28.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:29.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:29.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:29.632 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T12:09:29.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T12:09:29.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:29.633 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:29.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:30.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:30.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:30.821 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T12:09:30.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T12:09:30.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:30.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:31.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:32.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:32.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:32.008 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T12:09:32.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T12:09:32.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:32.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:32.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:33.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:33.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:33.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T12:09:33.196 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T12:09:33.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:33.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:33.381 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:34.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:34.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:34.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T12:09:34.383 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T12:09:34.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:34.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:34.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:35.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:35.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:35.571 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T12:09:35.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T12:09:35.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:35.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:35.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:36.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:36.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:36.751 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T12:09:36.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T12:09:36.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:36.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:36.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:37.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:37.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:37.940 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T12:09:37.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T12:09:37.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:37.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:38.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:39.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:39.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:39.125 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T12:09:39.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T12:09:39.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:39.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:39.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:40.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:40.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:40.309 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T12:09:40.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T12:09:40.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:40.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:40.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:41.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:41.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:41.490 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-18T12:09:41.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T12:09:41.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:41.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:41.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:42.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:42.677 INFO:tasks.workunit.client.0.smithi157.stdout:20 2024-09-18T12:09:42.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:42.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T12:09:42.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:42.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:42.865 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:43.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:43.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:43.868 INFO:tasks.workunit.client.0.smithi157.stdout:21 2024-09-18T12:09:43.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-18T12:09:43.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:43.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:44.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:45.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:45.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:45.082 INFO:tasks.workunit.client.0.smithi157.stdout:22 2024-09-18T12:09:45.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-18T12:09:45.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:45.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:09:45.422 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/198107,v1:127.0.0.1:6801/198107] [v2:127.0.0.1:6802/198107,v1:127.0.0.1:6803/198107] exists,up 65936d02-f832-40b7-9b7a-c5f99ff973ef 2024-09-18T12:09:45.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:09:45.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:09:45.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:09:45.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:125: TEST_check_version_health_2: run_osd td/ver-health 1 2024-09-18T12:09:45.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-18T12:09:45.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:09:45.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T12:09:45.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:09:45.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-09-18T12:09:45.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:09:45.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:09:45.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:09:45.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:09:45.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-18T12:09:45.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-18T12:09:45.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:09:45.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T12:09:45.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:09:45.428 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:09:45.428 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:09:45.428 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:09:45.428 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:09:45.428 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:09:45.428 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:09:45.428 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:09:45.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:09:45.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:09:45.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:09:45.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:09:45.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:09:45.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:09:45.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:09:45.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:09:45.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:09:45.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:09:45.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:09:45.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T12:09:45.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-09-18T12:09:45.429 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:09:45.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=55ad9da8-af52-4fe5-9ca8-cd4993f0ad4c 2024-09-18T12:09:45.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 55ad9da8-af52-4fe5-9ca8-cd4993f0ad4c' 2024-09-18T12:09:45.430 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 55ad9da8-af52-4fe5-9ca8-cd4993f0ad4c 2024-09-18T12:09:45.430 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:09:45.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAJw+pmo7IpGhAAAEt4DvM+cQdj1ZmxNDEbCw== 2024-09-18T12:09:45.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAJw+pmo7IpGhAAAEt4DvM+cQdj1ZmxNDEbCw=="}' 2024-09-18T12:09:45.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 55ad9da8-af52-4fe5-9ca8-cd4993f0ad4c -i td/ver-health/1/new.json 2024-09-18T12:09:45.887 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:09:45.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-09-18T12:09:45.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJw+pmo7IpGhAAAEt4DvM+cQdj1ZmxNDEbCw== --osd-uuid 55ad9da8-af52-4fe5-9ca8-cd4993f0ad4c 2024-09-18T12:09:45.921 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:45.920+0000 7f446f323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:09:45.923 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:45.921+0000 7f446f323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:09:45.924 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:45.922+0000 7f446f323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:09:45.924 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:45.923+0000 7f446f323440 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-18T12:09:45.925 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:45.923+0000 7f446f323440 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-18T12:09:45.925 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:45.923+0000 7f446f323440 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-18T12:09:46.013 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:46.011+0000 7f446f323440 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2024-09-18T12:09:51.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-09-18T12:09:51.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:09:51.777 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T12:09:51.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T12:09:51.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:09:51.969 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T12:09:51.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T12:09:51.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:09:51.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:09:51.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:09:51.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:09:51.991 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:51.990+0000 7f8c5ec1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:09:52.002 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:52.001+0000 7f8c5ec1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:09:52.003 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:52.002+0000 7f8c5ec1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:09:52.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T12:09:52.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:09:52.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T12:09:52.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:09:52.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:09:52.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:52.197 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:09:52.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:09:52.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:52.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:09:52.532 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:52.531+0000 7f8c5ec1f440 -1 Falling back to public interface 2024-09-18T12:09:52.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:53.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:53.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:53.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:09:53.578 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:09:53.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:53.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:09:53.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:54.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:54.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:54.765 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:09:54.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:09:54.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:54.766 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:09:54.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:55.040 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:09:55.038+0000 7f8c5ec1f440 -1 osd.1 0 log_to_monitors true 2024-09-18T12:09:55.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:55.950 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:09:55.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:55.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:09:55.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:55.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:09:56.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:57.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:57.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:57.159 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T12:09:57.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T12:09:57.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:57.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:09:57.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:58.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:58.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:58.345 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T12:09:58.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T12:09:58.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:58.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:09:58.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:09:59.583 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:09:59.583 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:09:59.584 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T12:09:59.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T12:09:59.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:09:59.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:09:59.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:00.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:00.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:00.771 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T12:10:00.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T12:10:00.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:00.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:10:00.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:01.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:01.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:01.957 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T12:10:01.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T12:10:01.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:01.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:10:02.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:03.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:03.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:03.143 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T12:10:03.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T12:10:03.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:03.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:10:03.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:04.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:04.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:04.331 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T12:10:04.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T12:10:04.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:04.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:10:04.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:05.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:05.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:05.518 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T12:10:05.518 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T12:10:05.518 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:05.518 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:10:05.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:06.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:06.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:06.700 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T12:10:06.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T12:10:06.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:06.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:10:06.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:07.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:07.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:07.888 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T12:10:07.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T12:10:07.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:07.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:10:08.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:09.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:09.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:09.076 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T12:10:09.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T12:10:09.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:09.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:10:09.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:10.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:10.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:10.264 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T12:10:10.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T12:10:10.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:10.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:10:10.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:11.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:11.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:11.446 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T12:10:11.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T12:10:11.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:11.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:10:11.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:12.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:12.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:12.903 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T12:10:12.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T12:10:12.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:12.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:10:13.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:14.090 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:14.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:14.091 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T12:10:14.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T12:10:14.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:14.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:10:14.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:15.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:15.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:15.700 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-18T12:10:15.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T12:10:15.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:15.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:10:15.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:16.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:16.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:16.888 INFO:tasks.workunit.client.0.smithi157.stdout:20 2024-09-18T12:10:16.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T12:10:16.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:16.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:10:17.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:18.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:18.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:18.075 INFO:tasks.workunit.client.0.smithi157.stdout:21 2024-09-18T12:10:18.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-18T12:10:18.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:18.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:10:18.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:19.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:19.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:19.261 INFO:tasks.workunit.client.0.smithi157.stdout:22 2024-09-18T12:10:19.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-18T12:10:19.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:19.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:10:19.515 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/199987,v1:127.0.0.1:6809/199987] [v2:127.0.0.1:6810/199987,v1:127.0.0.1:6811/199987] exists,up 55ad9da8-af52-4fe5-9ca8-cd4993f0ad4c 2024-09-18T12:10:19.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:10:19.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:10:19.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:10:19.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:126: TEST_check_version_health_2: run_osd td/ver-health 2 2024-09-18T12:10:19.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-18T12:10:19.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:10:19.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T12:10:19.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:10:19.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-09-18T12:10:19.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:10:19.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:10:19.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:10:19.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:10:19.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-18T12:10:19.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-18T12:10:19.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:10:19.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T12:10:19.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:10:19.520 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:10:19.521 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:10:19.521 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:10:19.521 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:10:19.521 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:10:19.521 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:10:19.521 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:10:19.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:10:19.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:10:19.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:10:19.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:10:19.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:10:19.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:10:19.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:10:19.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:10:19.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:10:19.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:10:19.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:10:19.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T12:10:19.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-09-18T12:10:19.522 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:10:19.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ca3de890-922a-4d58-908f-29dc42e39d63 2024-09-18T12:10:19.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ca3de890-922a-4d58-908f-29dc42e39d63' 2024-09-18T12:10:19.522 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 ca3de890-922a-4d58-908f-29dc42e39d63 2024-09-18T12:10:19.523 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:10:19.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQArw+pmMp+tHxAAlMwvSEYw/AgdcgxSufLq6w== 2024-09-18T12:10:19.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQArw+pmMp+tHxAAlMwvSEYw/AgdcgxSufLq6w=="}' 2024-09-18T12:10:19.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ca3de890-922a-4d58-908f-29dc42e39d63 -i td/ver-health/2/new.json 2024-09-18T12:10:19.928 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:10:19.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-09-18T12:10:19.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQArw+pmMp+tHxAAlMwvSEYw/AgdcgxSufLq6w== --osd-uuid ca3de890-922a-4d58-908f-29dc42e39d63 2024-09-18T12:10:19.961 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:10:19.960+0000 7f1147475440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:10:19.963 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:10:19.962+0000 7f1147475440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:10:19.964 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:10:19.963+0000 7f1147475440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:10:19.964 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:10:19.963+0000 7f1147475440 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-18T12:10:19.964 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:10:19.963+0000 7f1147475440 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-18T12:10:19.964 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:10:19.963+0000 7f1147475440 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-18T12:10:20.103 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:10:20.102+0000 7f1147475440 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2024-09-18T12:10:26.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-09-18T12:10:26.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:10:26.235 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-18T12:10:26.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T12:10:26.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:10:26.428 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-18T12:10:26.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T12:10:26.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:10:26.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:10:26.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:10:26.429 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:10:26.450 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:10:26.448+0000 7fc26b013440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:10:26.468 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:10:26.466+0000 7fc26b013440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:10:26.469 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:10:26.467+0000 7fc26b013440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:10:26.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T12:10:26.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:10:26.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T12:10:26.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:10:26.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:10:26.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:26.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:10:26.681 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:10:26.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:26.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:26.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:26.995 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:10:26.994+0000 7fc26b013440 -1 Falling back to public interface 2024-09-18T12:10:27.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:27.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:27.869 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:10:27.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:10:27.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:27.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:28.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:29.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:29.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:29.057 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:10:29.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:10:29.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:29.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:29.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:29.657 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:10:29.655+0000 7fc26b013440 -1 osd.2 0 log_to_monitors true 2024-09-18T12:10:30.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:30.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:30.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:10:30.246 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:10:30.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:30.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:30.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:31.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:31.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:31.435 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-18T12:10:31.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T12:10:31.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:31.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:31.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:32.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:32.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:32.622 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-18T12:10:32.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T12:10:32.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:32.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:32.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:33.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:33.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:33.809 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-18T12:10:33.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T12:10:33.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:33.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:33.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:34.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:34.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:34.996 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-18T12:10:34.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T12:10:34.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:34.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:35.181 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:36.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:36.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:36.182 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-18T12:10:36.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T12:10:36.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:36.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:36.368 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:37.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:37.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:37.370 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-18T12:10:37.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T12:10:37.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:37.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:37.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:38.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:38.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:38.556 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-18T12:10:38.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T12:10:38.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:38.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:38.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:39.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:39.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:39.741 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-18T12:10:39.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T12:10:39.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:39.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:39.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:40.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:40.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:40.929 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-18T12:10:40.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T12:10:40.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:40.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:41.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:42.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:42.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:42.117 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-18T12:10:42.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T12:10:42.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:42.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:42.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:43.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:43.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:43.307 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-18T12:10:43.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T12:10:43.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:43.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:43.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:44.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:44.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:44.494 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-18T12:10:44.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T12:10:44.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:44.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:44.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:45.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:45.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:45.679 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-18T12:10:45.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T12:10:45.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:45.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:45.865 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:46.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:46.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:46.867 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-18T12:10:46.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T12:10:46.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:46.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:47.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:48.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:48.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:48.053 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-18T12:10:48.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T12:10:48.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:48.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:48.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:49.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:49.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:49.238 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-18T12:10:49.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T12:10:49.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:49.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:49.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:50.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:50.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:50.427 INFO:tasks.workunit.client.0.smithi157.stdout:20 2024-09-18T12:10:50.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T12:10:50.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:50.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:50.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:51.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:51.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:51.609 INFO:tasks.workunit.client.0.smithi157.stdout:21 2024-09-18T12:10:51.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-18T12:10:51.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:51.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:51.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:10:52.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:10:52.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:10:52.796 INFO:tasks.workunit.client.0.smithi157.stdout:22 2024-09-18T12:10:52.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-18T12:10:52.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:10:52.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:10:53.282 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/201866,v1:127.0.0.1:6817/201866] [v2:127.0.0.1:6818/201866,v1:127.0.0.1:6819/201866] exists,up ca3de890-922a-4d58-908f-29dc42e39d63 2024-09-18T12:10:53.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:10:53.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:10:53.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:10:53.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:127: TEST_check_version_health_2: run_mgr td/ver-health x 2024-09-18T12:10:53.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ver-health 2024-09-18T12:10:53.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T12:10:53.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T12:10:53.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T12:10:53.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ver-health/x 2024-09-18T12:10:53.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T12:10:53.645 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T12:10:53.646 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:10:53.646 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:10:53.646 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:10:53.646 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:10:53.646 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:10:53.646 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:10:53.647 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T12:10:53.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/x '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T12:10:53.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:128: TEST_check_version_health_2: run_mgr td/ver-health y 2024-09-18T12:10:53.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ver-health 2024-09-18T12:10:53.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T12:10:53.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=y 2024-09-18T12:10:53.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T12:10:53.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ver-health/y 2024-09-18T12:10:53.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T12:10:53.899 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T12:10:53.899 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:10:53.900 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:10:53.900 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:10:53.900 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:10:53.900 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:10:53.900 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:10:53.900 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T12:10:53.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id y --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/y '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T12:10:53.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:129: TEST_check_version_health_2: run_mds td/ver-health m 2024-09-18T12:10:53.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-09-18T12:10:53.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-18T12:10:53.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=m 2024-09-18T12:10:53.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-18T12:10:53.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/m 2024-09-18T12:10:53.925 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-18T12:10:53.925 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:10:53.925 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:10:53.925 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:10:53.925 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:10:53.925 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:10:53.926 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:10:53.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-09-18T12:10:53.978 INFO:tasks.workunit.client.0.smithi157.stdout:starting mds.m at 2024-09-18T12:10:53.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:130: TEST_check_version_health_2: run_mds td/ver-health n 2024-09-18T12:10:53.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-09-18T12:10:53.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-18T12:10:53.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=n 2024-09-18T12:10:53.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-18T12:10:53.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/n 2024-09-18T12:10:53.983 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-18T12:10:53.983 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:10:53.983 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:10:53.983 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:10:53.983 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:10:53.983 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:10:53.983 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:10:53.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id n --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/n '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-09-18T12:10:54.004 INFO:tasks.workunit.client.0.smithi157.stdout:starting mds.n at 2024-09-18T12:10:54.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:132: TEST_check_version_health_2: sleep 5 2024-09-18T12:10:59.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:133: TEST_check_version_health_2: ceph health detail 2024-09-18T12:10:59.486 INFO:tasks.workunit.client.0.smithi157.stdout:HEALTH_OK 2024-09-18T12:10:59.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: ceph health detail 2024-09-18T12:10:59.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: grep DAEMON_OLD_VERSION 2024-09-18T12:11:00.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:137: TEST_check_version_health_2: kill_daemons td/ver-health KILL mon.b 2024-09-18T12:11:00.074 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:11:00.074 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:11:00.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:11:00.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:11:00.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:11:00.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:11:00.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-18T12:11:00.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: run_mon td/ver-health b --mon_warn_older_version_delay=0 2024-09-18T12:11:00.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-18T12:11:00.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T12:11:00.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-18T12:11:00.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T12:11:00.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-09-18T12:11:00.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --mon_warn_older_version_delay=0 2024-09-18T12:11:00.197 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:00.195+0000 7f4d6090c900 -1 'td/ver-health/b' already exists and is not empty: monitor may already exist 2024-09-18T12:11:00.199 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T12:11:00.199 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:11:00.199 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:11:00.199 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:11:00.199 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:00.199 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:00.200 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:11:00.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_warn_older_version_delay=0 2024-09-18T12:11:00.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T12:11:00.463 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-18T12:11:00.463 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:11:00.463 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-18T12:11:00.463 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T12:11:00.463 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T12:11:00.464 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-18T12:11:00.464 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-18T12:11:00.464 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-18T12:11:00.464 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:11:00.464 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:00.464 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:00.464 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188934/ceph-mon.b.asok 2024-09-18T12:11:00.464 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:11:00.464 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188934/ceph-mon.b.asok config get fsid 2024-09-18T12:11:00.536 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-18T12:11:00.536 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:11:00.536 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-18T12:11:00.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T12:11:00.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T12:11:00.537 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-18T12:11:00.537 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-18T12:11:00.537 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-18T12:11:00.537 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:11:00.537 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:00.537 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:00.537 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188934/ceph-mon.b.asok 2024-09-18T12:11:00.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:11:00.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188934/ceph-mon.b.asok config get mon_host 2024-09-18T12:11:00.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:142: TEST_check_version_health_2: kill_daemons td/ver-health KILL mds.m 2024-09-18T12:11:00.610 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:11:00.610 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:11:00.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:11:00.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:11:00.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:11:00.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:11:00.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-18T12:11:00.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: run_mds td/ver-health m 2024-09-18T12:11:00.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-09-18T12:11:00.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-18T12:11:00.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=m 2024-09-18T12:11:00.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-18T12:11:00.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/m 2024-09-18T12:11:00.717 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-18T12:11:00.717 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:11:00.717 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:11:00.717 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:11:00.717 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:00.717 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:00.717 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:11:00.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-09-18T12:11:00.937 INFO:tasks.workunit.client.0.smithi157.stdout:starting mds.m at 2024-09-18T12:11:00.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:145: TEST_check_version_health_2: wait_for_health_string 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-18T12:11:00.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-18T12:11:00.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-18T12:11:00.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-18T12:11:00.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-18T12:11:00.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-18T12:11:00.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:11:00.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:11:00.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-18T12:11:01.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:11:02.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:11:02.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:11:02.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:11:02.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-18T12:11:02.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:11:03.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:11:03.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:11:03.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:11:03.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-18T12:11:04.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:11:05.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:11:05.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:11:05.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:11:05.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-18T12:11:06.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:11:07.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:11:07.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:11:07.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:11:07.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-18T12:11:07.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:11:08.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:11:08.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:11:08.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:11:08.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-18T12:11:08.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:11:09.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:11:09.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:11:09.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:11:09.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-18T12:11:10.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:11:11.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:11:11.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:11:11.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:11:11.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-18T12:11:11.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-18T12:11:11.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-18T12:11:11.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-18T12:11:11.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-18T12:11:11.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-18T12:11:11.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:147: TEST_check_version_health_2: ceph health detail 2024-09-18T12:11:12.375 INFO:tasks.workunit.client.0.smithi157.stdout:HEALTH_WARN There are daemons running an older version of ceph 2024-09-18T12:11:12.375 INFO:tasks.workunit.client.0.smithi157.stdout:[WRN] DAEMON_OLD_VERSION: There are daemons running an older version of ceph 2024-09-18T12:11:12.375 INFO:tasks.workunit.client.0.smithi157.stdout: mon.b mds.m are running an older version of ceph: 01.00.00-gversion-test 2024-09-18T12:11:12.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: ceph health detail 2024-09-18T12:11:12.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-18T12:11:12.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: ceph health detail 2024-09-18T12:11:12.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There are daemons running an older version of ceph' 2024-09-18T12:11:13.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: ceph health detail 2024-09-18T12:11:13.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: grep -q 'mon.b mds.m are running an older version of ceph: 01.00.00-gversion-test' 2024-09-18T12:11:13.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:153: TEST_check_version_health_2: kill_daemons td/ver-health KILL osd.2 2024-09-18T12:11:13.938 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:11:13.938 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:11:13.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:11:13.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:11:13.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:11:14.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:11:14.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:154: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-18T12:11:14.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:154: TEST_check_version_health_2: activate_osd td/ver-health 2 2024-09-18T12:11:14.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-18T12:11:14.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T12:11:14.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T12:11:14.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T12:11:14.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2024-09-18T12:11:14.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:11:14.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:11:14.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:11:14.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:11:14.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-18T12:11:14.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-18T12:11:14.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T12:11:14.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T12:11:14.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:11:14.047 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T12:11:14.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:11:14.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:11:14.048 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:11:14.048 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:14.048 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:14.048 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:11:14.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:11:14.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:11:14.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:11:14.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:11:14.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:11:14.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:11:14.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:11:14.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:11:14.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T12:11:14.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T12:11:14.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2024-09-18T12:11:14.049 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-18T12:11:14.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T12:11:14.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:11:14.050 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2024-09-18T12:11:14.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T12:11:14.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T12:11:14.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T12:11:14.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T12:11:14.067 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:14.065+0000 7f7c78f31440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:14.152 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:14.150+0000 7f7c78f31440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:14.153 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:14.152+0000 7f7c78f31440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:14.671 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:14.670+0000 7f7c78f31440 -1 Falling back to public interface 2024-09-18T12:11:14.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T12:11:14.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:11:14.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T12:11:14.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:11:14.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:11:14.710 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:11:14.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:11:14.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:11:14.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:11:14.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:11:15.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:11:16.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:11:16.121 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:11:16.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:11:16.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:11:16.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:11:16.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:11:16.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:11:17.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:11:17.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:11:17.511 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:11:17.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:11:17.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:11:17.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:11:17.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:11:18.017 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:18.016+0000 7f7c78f31440 -1 osd.2 17 log_to_monitors true 2024-09-18T12:11:18.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:11:18.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:11:18.870 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:11:18.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:11:18.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:11:18.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:11:19.285 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 18 last_clean_interval [15,17) [v2:127.0.0.1:6816/203658,v1:127.0.0.1:6817/203658] [v2:127.0.0.1:6818/203658,v1:127.0.0.1:6819/203658] exists,up ca3de890-922a-4d58-908f-29dc42e39d63 2024-09-18T12:11:19.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:11:19.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:11:19.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:11:19.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:155: TEST_check_version_health_2: kill_daemons td/ver-health KILL osd.0 2024-09-18T12:11:19.286 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:11:19.286 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:11:19.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:11:19.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:11:19.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:11:19.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:11:19.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:156: TEST_check_version_health_2: ceph_debug_version_for_testing=02.00.00-gversion-test 2024-09-18T12:11:19.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:156: TEST_check_version_health_2: activate_osd td/ver-health 0 2024-09-18T12:11:19.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-18T12:11:19.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T12:11:19.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T12:11:19.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T12:11:19.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2024-09-18T12:11:19.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:11:19.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:11:19.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:11:19.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:11:19.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-18T12:11:19.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-18T12:11:19.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T12:11:19.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T12:11:19.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:11:19.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T12:11:19.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:11:19.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:11:19.395 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:11:19.395 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:19.395 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:19.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:11:19.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:11:19.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:11:19.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:11:19.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:11:19.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:11:19.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:11:19.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:11:19.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:11:19.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T12:11:19.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T12:11:19.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2024-09-18T12:11:19.396 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T12:11:19.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T12:11:19.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:11:19.397 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2024-09-18T12:11:19.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T12:11:19.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T12:11:19.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T12:11:19.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T12:11:19.415 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:19.413+0000 7ff87041a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:19.435 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:19.433+0000 7ff87041a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:19.436 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:19.434+0000 7ff87041a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:19.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T12:11:19.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:11:19.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T12:11:19.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:11:19.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:11:19.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:11:19.815 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:11:19.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:11:19.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:11:19.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:11:19.952 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:19.951+0000 7ff87041a440 -1 Falling back to public interface 2024-09-18T12:11:20.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:11:21.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:11:21.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:11:21.231 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:11:21.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:11:21.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:11:21.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:11:21.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:11:22.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:11:22.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:11:22.592 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:11:22.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T12:11:22.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:11:22.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:11:22.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:11:23.148 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:23.147+0000 7ff87041a440 -1 osd.0 18 log_to_monitors true 2024-09-18T12:11:23.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:11:23.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:11:23.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T12:11:23.968 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-18T12:11:23.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:11:23.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:11:24.405 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 21 last_clean_interval [5,18) [v2:127.0.0.1:6800/204228,v1:127.0.0.1:6801/204228] [v2:127.0.0.1:6802/204228,v1:127.0.0.1:6803/204228] exists,up 65936d02-f832-40b7-9b7a-c5f99ff973ef 2024-09-18T12:11:24.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:11:24.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:11:24.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:11:24.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:158: TEST_check_version_health_2: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-18T12:11:24.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-18T12:11:24.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-18T12:11:24.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-18T12:11:24.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-18T12:11:24.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-18T12:11:24.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:11:24.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:11:24.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-18T12:11:24.865 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:11:25.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:11:25.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:11:25.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:11:25.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-18T12:11:26.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-18T12:11:26.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-18T12:11:26.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-18T12:11:26.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-18T12:11:26.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-18T12:11:26.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:160: TEST_check_version_health_2: ceph health detail 2024-09-18T12:11:26.822 INFO:tasks.workunit.client.0.smithi157.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph 2024-09-18T12:11:26.822 INFO:tasks.workunit.client.0.smithi157.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2024-09-18T12:11:26.822 INFO:tasks.workunit.client.0.smithi157.stdout: mon.b osd.2 mds.m are running an older version of ceph: 01.00.00-gversion-test 2024-09-18T12:11:26.822 INFO:tasks.workunit.client.0.smithi157.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2024-09-18T12:11:26.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:161: TEST_check_version_health_2: ceph health 2024-09-18T12:11:26.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:161: TEST_check_version_health_2: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-18T12:11:27.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:162: TEST_check_version_health_2: ceph health detail 2024-09-18T12:11:27.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:162: TEST_check_version_health_2: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-18T12:11:27.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:163: TEST_check_version_health_2: ceph health detail 2024-09-18T12:11:27.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:163: TEST_check_version_health_2: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2024-09-18T12:11:28.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:164: TEST_check_version_health_2: ceph health detail 2024-09-18T12:11:28.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:164: TEST_check_version_health_2: grep -q 'mon.b osd.2 mds.m are running an older version of ceph: 01.00.00-gversion-test' 2024-09-18T12:11:28.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:165: TEST_check_version_health_2: ceph health detail 2024-09-18T12:11:28.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:165: TEST_check_version_health_2: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2024-09-18T12:11:29.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2024-09-18T12:11:29.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-18T12:11:29.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:11:29.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-18T12:11:29.422 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:11:29.422 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:11:29.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:11:29.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:11:29.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:11:29.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:11:29.533 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:11:29.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:11:29.533 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:11:29.535 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:11:29.535 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:11:29.535 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:11:29.535 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:11:29.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:11:29.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:11:29.536 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:11:29.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:11:29.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:11:29.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-18T12:11:29.554 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:11:29.555 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:29.555 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:29.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188934 2024-09-18T12:11:29.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:11:29.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:11:29.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-09-18T12:11:29.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-09-18T12:11:29.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-18T12:11:29.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-18T12:11:29.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-18T12:11:29.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:11:29.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-18T12:11:29.557 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:11:29.557 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:11:29.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:11:29.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:11:29.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:11:29.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:11:29.558 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:11:29.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:11:29.559 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:11:29.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:11:29.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:11:29.560 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:11:29.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:11:29.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:11:29.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:11:29.562 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:11:29.562 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:11:29.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:11:29.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-18T12:11:29.564 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:11:29.564 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:29.564 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:29.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188934 2024-09-18T12:11:29.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:11:29.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:11:29.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-18T12:11:29.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T12:11:29.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:29.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:29.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.188934 2024-09-18T12:11:29.569 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T12:11:29.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T12:11:29.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T12:11:29.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-18T12:11:29.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_3 td/ver-health 2024-09-18T12:11:29.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:170: TEST_check_version_health_3: local dir=td/ver-health 2024-09-18T12:11:29.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:173: TEST_check_version_health_3: CEPH_ARGS='--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:11:29.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:175: TEST_check_version_health_3: setup td/ver-health 2024-09-18T12:11:29.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-18T12:11:29.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-18T12:11:29.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-18T12:11:29.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:11:29.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-18T12:11:29.570 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:11:29.570 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:11:29.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:11:29.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:11:29.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:11:29.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:11:29.571 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:11:29.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:11:29.571 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:11:29.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:11:29.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:11:29.572 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:11:29.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:11:29.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:11:29.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:11:29.574 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:11:29.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:11:29.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:11:29.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-18T12:11:29.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:11:29.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:29.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:29.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188934 2024-09-18T12:11:29.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:11:29.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:11:29.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-18T12:11:29.578 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T12:11:29.579 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:29.579 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:29.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.188934 2024-09-18T12:11:29.581 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T12:11:29.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T12:11:29.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T12:11:29.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-18T12:11:29.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:178: TEST_check_version_health_3: run_mon td/ver-health a --public-addr=127.0.0.1:7165 2024-09-18T12:11:29.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-18T12:11:29.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T12:11:29.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T12:11:29.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T12:11:29.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-09-18T12:11:29.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 2024-09-18T12:11:29.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T12:11:29.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:11:29.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:11:29.855 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:11:29.855 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:29.855 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:29.855 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:11:29.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 2024-09-18T12:11:30.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T12:11:30.105 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T12:11:30.105 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:11:30.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T12:11:30.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T12:11:30.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T12:11:30.106 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T12:11:30.106 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T12:11:30.106 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T12:11:30.106 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:11:30.106 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:30.106 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:30.106 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188934/ceph-mon.a.asok 2024-09-18T12:11:30.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:11:30.107 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188934/ceph-mon.a.asok config get fsid 2024-09-18T12:11:30.180 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T12:11:30.181 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:11:30.181 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T12:11:30.181 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T12:11:30.181 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T12:11:30.181 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T12:11:30.181 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T12:11:30.181 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T12:11:30.181 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:11:30.181 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:30.181 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:30.181 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188934/ceph-mon.a.asok 2024-09-18T12:11:30.181 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:11:30.182 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188934/ceph-mon.a.asok config get mon_host 2024-09-18T12:11:30.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:179: TEST_check_version_health_3: run_mon td/ver-health b --public-addr=127.0.0.1:7166 2024-09-18T12:11:30.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-18T12:11:30.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T12:11:30.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-18T12:11:30.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T12:11:30.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-09-18T12:11:30.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 2024-09-18T12:11:30.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T12:11:30.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:11:30.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:11:30.845 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:11:30.846 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:30.846 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:30.846 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:11:30.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 2024-09-18T12:11:31.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T12:11:31.236 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-18T12:11:31.236 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:11:31.237 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-18T12:11:31.237 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T12:11:31.238 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T12:11:31.238 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-18T12:11:31.238 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-18T12:11:31.238 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-18T12:11:31.239 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:11:31.239 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:31.239 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:31.239 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188934/ceph-mon.b.asok 2024-09-18T12:11:31.239 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:11:31.239 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188934/ceph-mon.b.asok config get fsid 2024-09-18T12:11:31.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-18T12:11:31.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T12:11:31.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-18T12:11:31.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T12:11:31.324 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T12:11:31.324 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-18T12:11:31.324 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-18T12:11:31.324 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-18T12:11:31.324 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T12:11:31.324 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:31.324 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:31.324 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188934/ceph-mon.b.asok 2024-09-18T12:11:31.324 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T12:11:31.324 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188934/ceph-mon.b.asok config get mon_host 2024-09-18T12:11:31.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:181: TEST_check_version_health_3: local start_osd_time=286 2024-09-18T12:11:31.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:183: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2024-09-18T12:11:31.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:183: TEST_check_version_health_3: run_osd td/ver-health 0 2024-09-18T12:11:31.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-18T12:11:31.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:11:31.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T12:11:31.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:11:31.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-09-18T12:11:31.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:11:31.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:11:31.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:11:31.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:11:31.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-18T12:11:31.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-18T12:11:31.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:11:31.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-objectstore=memstore' 2024-09-18T12:11:31.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:11:31.405 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:11:31.405 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:11:31.405 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:11:31.405 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:11:31.405 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:31.405 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:31.406 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:11:31.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:11:31.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:11:31.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:11:31.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:11:31.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:11:31.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:11:31.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:11:31.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:11:31.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:11:31.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:11:31.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:11:31.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T12:11:31.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-09-18T12:11:31.407 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:11:31.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b4a26b0e-6dfe-4979-8918-031eeecda80b 2024-09-18T12:11:31.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b4a26b0e-6dfe-4979-8918-031eeecda80b' 2024-09-18T12:11:31.408 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 b4a26b0e-6dfe-4979-8918-031eeecda80b 2024-09-18T12:11:31.408 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:11:31.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBzw+pmkOS2GBAAeMvwY+aZC0J8ilzAeg1gLA== 2024-09-18T12:11:31.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBzw+pmkOS2GBAAeMvwY+aZC0J8ilzAeg1gLA=="}' 2024-09-18T12:11:31.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b4a26b0e-6dfe-4979-8918-031eeecda80b -i td/ver-health/0/new.json 2024-09-18T12:11:37.706 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:11:37.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-09-18T12:11:37.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBzw+pmkOS2GBAAeMvwY+aZC0J8ilzAeg1gLA== --osd-uuid b4a26b0e-6dfe-4979-8918-031eeecda80b 2024-09-18T12:11:37.741 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:37.739+0000 7fb4dde43440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:37.795 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:37.794+0000 7fb4dde43440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:37.796 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:37.795+0000 7fb4dde43440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:37.893 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:37.891+0000 7fb4dde43440 -1 memstore(td/ver-health/0) td/ver-health/0 2024-09-18T12:11:38.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-09-18T12:11:38.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:11:38.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T12:11:38.351 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-18T12:11:38.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:11:38.538 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T12:11:38.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T12:11:38.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:11:38.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:11:38.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:11:38.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:11:38.560 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:38.559+0000 7f3fb3121440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:38.572 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:38.570+0000 7f3fb3121440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:38.573 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:38.571+0000 7f3fb3121440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:38.574 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:38.573+0000 7f3fb3121440 -1 Falling back to public interface 2024-09-18T12:11:38.582 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:38.581+0000 7f3fb3121440 -1 osd.0 0 log_to_monitors true 2024-09-18T12:11:38.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T12:11:38.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:11:38.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T12:11:38.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:11:38.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:11:38.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:11:38.793 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:11:38.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:11:38.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:11:38.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:11:39.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:11:40.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:11:40.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:11:40.061 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:11:40.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:11:40.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:11:40.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:11:40.246 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/205308,v1:127.0.0.1:6801/205308] [v2:127.0.0.1:6802/205308,v1:127.0.0.1:6803/205308] exists,up b4a26b0e-6dfe-4979-8918-031eeecda80b 2024-09-18T12:11:40.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:11:40.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:11:40.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:11:40.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:184: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2024-09-18T12:11:40.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:184: TEST_check_version_health_3: run_osd td/ver-health 1 2024-09-18T12:11:40.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-18T12:11:40.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:11:40.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T12:11:40.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:11:40.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-09-18T12:11:40.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:11:40.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:11:40.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:11:40.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:11:40.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-18T12:11:40.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-18T12:11:40.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:11:40.252 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-objectstore=memstore' 2024-09-18T12:11:40.252 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:11:40.252 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:11:40.252 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:11:40.252 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:11:40.252 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:11:40.252 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:40.252 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:40.252 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:11:40.252 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:11:40.252 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:11:40.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:11:40.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:11:40.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:11:40.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:11:40.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:11:40.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:11:40.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:11:40.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:11:40.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:11:40.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T12:11:40.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-09-18T12:11:40.253 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:11:40.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d8ed87ea-9af6-49e5-bb32-0f18bee46c71 2024-09-18T12:11:40.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d8ed87ea-9af6-49e5-bb32-0f18bee46c71' 2024-09-18T12:11:40.254 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 d8ed87ea-9af6-49e5-bb32-0f18bee46c71 2024-09-18T12:11:40.254 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:11:40.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB8w+pmL+m1DxAAH2WgK8zvh/5A0nzbd0llTw== 2024-09-18T12:11:40.265 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB8w+pmL+m1DxAAH2WgK8zvh/5A0nzbd0llTw=="}' 2024-09-18T12:11:40.265 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d8ed87ea-9af6-49e5-bb32-0f18bee46c71 -i td/ver-health/1/new.json 2024-09-18T12:11:40.598 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:11:40.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-09-18T12:11:40.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8w+pmL+m1DxAAH2WgK8zvh/5A0nzbd0llTw== --osd-uuid d8ed87ea-9af6-49e5-bb32-0f18bee46c71 2024-09-18T12:11:40.633 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:40.631+0000 7fc28aa11440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:40.634 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:40.633+0000 7fc28aa11440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:40.635 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:40.634+0000 7fc28aa11440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:40.708 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:40.707+0000 7fc28aa11440 -1 memstore(td/ver-health/1) td/ver-health/1 2024-09-18T12:11:41.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-09-18T12:11:41.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:11:41.299 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-18T12:11:41.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T12:11:41.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:11:41.482 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T12:11:41.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T12:11:41.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:11:41.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:11:41.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:11:41.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:11:41.505 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:41.504+0000 7fce4328b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:41.512 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:41.511+0000 7fce4328b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:41.513 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:41.512+0000 7fce4328b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:41.514 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:41.513+0000 7fce4328b440 -1 Falling back to public interface 2024-09-18T12:11:41.524 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:41.523+0000 7fce4328b440 -1 osd.1 0 log_to_monitors true 2024-09-18T12:11:41.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T12:11:41.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:11:41.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T12:11:41.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:11:41.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:11:41.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:11:41.736 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:11:41.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:11:41.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:11:41.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:11:42.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:11:43.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:11:43.361 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:11:43.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:11:43.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:11:43.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:11:43.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:11:43.537 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/205513,v1:127.0.0.1:6809/205513] [v2:127.0.0.1:6810/205513,v1:127.0.0.1:6811/205513] exists,up d8ed87ea-9af6-49e5-bb32-0f18bee46c71 2024-09-18T12:11:43.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:11:43.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:11:43.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:11:43.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:185: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2024-09-18T12:11:43.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:185: TEST_check_version_health_3: run_osd td/ver-health 2 2024-09-18T12:11:43.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-18T12:11:43.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T12:11:43.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T12:11:43.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T12:11:43.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-09-18T12:11:43.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:11:43.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:11:43.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:11:43.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:11:43.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-18T12:11:43.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-18T12:11:43.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T12:11:43.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-objectstore=memstore' 2024-09-18T12:11:43.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:11:43.542 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T12:11:43.542 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:11:43.542 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:11:43.542 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:11:43.542 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:43.542 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:43.542 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:11:43.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:11:43.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:11:43.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T12:11:43.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T12:11:43.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:11:43.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:11:43.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:11:43.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:11:43.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:11:43.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:11:43.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T12:11:43.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T12:11:43.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-09-18T12:11:43.543 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T12:11:43.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0a60afd5-ba4b-4fdf-ab0c-d6712d19fedc 2024-09-18T12:11:43.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0a60afd5-ba4b-4fdf-ab0c-d6712d19fedc' 2024-09-18T12:11:43.544 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 0a60afd5-ba4b-4fdf-ab0c-d6712d19fedc 2024-09-18T12:11:43.544 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T12:11:43.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB/w+pmpyT8IBAArrsaaNOgnaHT1eujPPIBgQ== 2024-09-18T12:11:43.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB/w+pmpyT8IBAArrsaaNOgnaHT1eujPPIBgQ=="}' 2024-09-18T12:11:43.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0a60afd5-ba4b-4fdf-ab0c-d6712d19fedc -i td/ver-health/2/new.json 2024-09-18T12:11:44.023 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-18T12:11:44.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-09-18T12:11:44.033 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB/w+pmpyT8IBAArrsaaNOgnaHT1eujPPIBgQ== --osd-uuid 0a60afd5-ba4b-4fdf-ab0c-d6712d19fedc 2024-09-18T12:11:44.057 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:44.056+0000 7fa115e69440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:44.059 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:44.058+0000 7fa115e69440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:44.060 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:44.059+0000 7fa115e69440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:44.140 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:44.139+0000 7fa115e69440 -1 memstore(td/ver-health/2) td/ver-health/2 2024-09-18T12:11:44.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-09-18T12:11:44.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T12:11:44.775 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-18T12:11:44.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T12:11:44.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T12:11:44.963 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-18T12:11:44.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T12:11:44.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:11:44.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T12:11:44.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T12:11:44.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T12:11:44.986 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:44.984+0000 7fc6fe766440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:44.995 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:44.994+0000 7fc6fe766440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:44.997 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:44.996+0000 7fc6fe766440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:44.998 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:44.997+0000 7fc6fe766440 -1 Falling back to public interface 2024-09-18T12:11:45.009 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:45.007+0000 7fc6fe766440 -1 osd.2 0 log_to_monitors true 2024-09-18T12:11:45.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T12:11:45.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:11:45.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T12:11:45.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:11:45.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:11:45.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:11:45.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:11:45.191 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:11:45.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:11:45.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:11:45.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:11:46.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:11:46.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:11:46.483 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:11:46.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:11:46.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:11:46.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:11:46.726 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/205718,v1:127.0.0.1:6817/205718] [v2:127.0.0.1:6818/205718,v1:127.0.0.1:6819/205718] exists,up 0a60afd5-ba4b-4fdf-ab0c-d6712d19fedc 2024-09-18T12:11:46.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:11:46.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:11:46.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:11:46.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:187: TEST_check_version_health_3: local warn_older_version_delay=35 2024-09-18T12:11:46.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:189: TEST_check_version_health_3: sleep 5 2024-09-18T12:11:51.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:190: TEST_check_version_health_3: ceph health detail 2024-09-18T12:11:51.906 INFO:tasks.workunit.client.0.smithi157.stdout:HEALTH_OK 2024-09-18T12:11:51.916 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:192: TEST_check_version_health_3: ceph health detail 2024-09-18T12:11:51.916 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:192: TEST_check_version_health_3: grep DAEMON_OLD_VERSION 2024-09-18T12:11:52.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:193: TEST_check_version_health_3: ceph tell 'mon.*' injectargs '--mon_warn_older_version_delay 35' 2024-09-18T12:11:52.300 INFO:tasks.workunit.client.0.smithi157.stdout:mon.a: {} 2024-09-18T12:11:52.313 INFO:tasks.workunit.client.0.smithi157.stdout:mon.b: {} 2024-09-18T12:11:52.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:194: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.1 2024-09-18T12:11:52.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:11:52.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:11:52.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:11:52.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:11:52.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:11:52.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:11:52.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2024-09-18T12:11:52.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-18T12:11:52.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: activate_osd td/ver-health 1 2024-09-18T12:11:52.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-18T12:11:52.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T12:11:52.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T12:11:52.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T12:11:52.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/1 2024-09-18T12:11:52.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:11:52.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:11:52.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:11:52.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:11:52.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-18T12:11:52.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-18T12:11:52.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T12:11:52.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-objectstore=memstore' 2024-09-18T12:11:52.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:11:52.437 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T12:11:52.437 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:11:52.437 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:11:52.437 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:11:52.438 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:11:52.438 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:11:52.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:11:52.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:11:52.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:11:52.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:11:52.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:11:52.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:11:52.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:11:52.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:11:52.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:11:52.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T12:11:52.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T12:11:52.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/1 2024-09-18T12:11:52.440 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-18T12:11:52.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T12:11:52.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:11:52.442 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/1/whoami 2024-09-18T12:11:52.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T12:11:52.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T12:11:52.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T12:11:52.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T12:11:52.453 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:52.452+0000 7fb36fa09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:52.486 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:52.485+0000 7fb36fa09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:52.487 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:52.486+0000 7fb36fa09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:11:52.488 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:52.487+0000 7fb36fa09440 -1 Falling back to public interface 2024-09-18T12:11:52.498 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:11:52.497+0000 7fb36fa09440 -1 osd.1 0 log_to_monitors true 2024-09-18T12:11:52.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T12:11:52.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:11:52.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T12:11:52.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:11:52.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:11:52.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:11:52.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:11:52.669 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:11:52.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:11:52.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:11:52.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:11:53.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:11:53.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:11:53.983 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:11:53.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:11:53.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:11:53.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T12:11:54.160 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [0,0) [v2:127.0.0.1:6808/205941,v1:127.0.0.1:6809/205941] [v2:127.0.0.1:6810/205941,v1:127.0.0.1:6811/205941] exists,up d8ed87ea-9af6-49e5-bb32-0f18bee46c71 2024-09-18T12:11:54.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:11:54.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:11:54.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:11:54.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:200: TEST_check_version_health_3: sleep 10 2024-09-18T12:12:04.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:202: TEST_check_version_health_3: ceph health detail 2024-09-18T12:12:04.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:202: TEST_check_version_health_3: grep DAEMON_OLD_VERSION 2024-09-18T12:12:04.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:205: TEST_check_version_health_3: wait_for_health_string 'HEALTH_WARN .*There is a daemon running an older version of ceph' 20 2024-09-18T12:12:04.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:12:04.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-18T12:12:04.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-18T12:12:04.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-18T12:12:04.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-18T12:12:04.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:12:04.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:12:04.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:12:04.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:12:05.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:12:05.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:12:05.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:12:05.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:12:05.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:12:06.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:12:06.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:12:06.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:12:06.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:12:06.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:12:07.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:12:07.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:12:07.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:12:07.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:12:08.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:12:09.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:12:09.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:12:09.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:12:09.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:12:09.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:12:10.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:12:10.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:12:10.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:12:10.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:12:10.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:12:11.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:12:11.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:12:11.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:12:11.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:12:11.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:12:12.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:12:12.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:12:12.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:12:12.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:12:12.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:12:13.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:12:13.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:12:13.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:12:13.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:12:14.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:12:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:12:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:12:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:12:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:12:15.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-18T12:12:15.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-18T12:12:15.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-18T12:12:15.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-18T12:12:15.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-18T12:12:15.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:207: TEST_check_version_health_3: ceph health detail 2024-09-18T12:12:15.562 INFO:tasks.workunit.client.0.smithi157.stdout:HEALTH_WARN There is a daemon running an older version of ceph 2024-09-18T12:12:15.562 INFO:tasks.workunit.client.0.smithi157.stdout:[WRN] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph 2024-09-18T12:12:15.562 INFO:tasks.workunit.client.0.smithi157.stdout: osd.1 is running an older version of ceph: 01.00.00-gversion-test 2024-09-18T12:12:15.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:209: TEST_check_version_health_3: ceph health detail 2024-09-18T12:12:15.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:209: TEST_check_version_health_3: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-18T12:12:15.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:210: TEST_check_version_health_3: ceph health detail 2024-09-18T12:12:15.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:210: TEST_check_version_health_3: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph' 2024-09-18T12:12:16.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:211: TEST_check_version_health_3: ceph health detail 2024-09-18T12:12:16.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:211: TEST_check_version_health_3: grep -q 'osd.1 is running an older version of ceph: 01.00.00-gversion-test' 2024-09-18T12:12:16.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:213: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.2 2024-09-18T12:12:16.329 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:12:16.329 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:12:16.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:12:16.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:12:16.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:12:16.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:12:16.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:214: TEST_check_version_health_3: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-18T12:12:16.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:214: TEST_check_version_health_3: activate_osd td/ver-health 2 2024-09-18T12:12:16.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-18T12:12:16.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T12:12:16.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T12:12:16.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T12:12:16.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2024-09-18T12:12:16.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:12:16.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:12:16.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:12:16.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:12:16.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-18T12:12:16.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-18T12:12:16.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T12:12:16.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T12:12:16.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:12:16.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T12:12:16.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:12:16.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:12:16.438 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:12:16.438 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:12:16.438 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:12:16.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:12:16.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:12:16.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:12:16.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:12:16.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:12:16.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:12:16.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:12:16.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:12:16.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:12:16.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T12:12:16.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T12:12:16.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2024-09-18T12:12:16.440 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-18T12:12:16.440 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T12:12:16.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:12:16.441 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2024-09-18T12:12:16.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T12:12:16.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T12:12:16.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T12:12:16.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T12:12:16.458 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:12:16.457+0000 7f4c5651d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:12:16.461 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:12:16.459+0000 7f4c5651d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:12:16.461 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:12:16.460+0000 7f4c5651d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:12:16.463 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:12:16.461+0000 7f4c5651d440 -1 Falling back to public interface 2024-09-18T12:12:16.473 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:12:16.472+0000 7f4c5651d440 -1 osd.2 0 log_to_monitors true 2024-09-18T12:12:16.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T12:12:16.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:12:16.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T12:12:16.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:12:16.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:12:16.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:12:16.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:12:16.816 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:12:16.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:12:16.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:12:17.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:12:18.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:12:18.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:12:18.068 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:12:18.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:12:18.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:12:18.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T12:12:18.252 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 21 last_clean_interval [0,0) [v2:127.0.0.1:6816/206458,v1:127.0.0.1:6817/206458] [v2:127.0.0.1:6818/206458,v1:127.0.0.1:6819/206458] exists,up 0a60afd5-ba4b-4fdf-ab0c-d6712d19fedc 2024-09-18T12:12:18.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:12:18.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:12:18.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:12:18.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:215: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.0 2024-09-18T12:12:18.253 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:12:18.253 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:12:18.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:12:18.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:12:18.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:12:18.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:12:18.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:216: TEST_check_version_health_3: ceph_debug_version_for_testing=02.00.00-gversion-test 2024-09-18T12:12:18.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:216: TEST_check_version_health_3: activate_osd td/ver-health 0 2024-09-18T12:12:18.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-18T12:12:18.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T12:12:18.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T12:12:18.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T12:12:18.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2024-09-18T12:12:18.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-18T12:12:18.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T12:12:18.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T12:12:18.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T12:12:18.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-18T12:12:18.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-18T12:12:18.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T12:12:18.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T12:12:18.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-18T12:12:18.363 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T12:12:18.363 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T12:12:18.363 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T12:12:18.363 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T12:12:18.363 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:12:18.363 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:12:18.363 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:12:18.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' 2024-09-18T12:12:18.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T12:12:18.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-18T12:12:18.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-18T12:12:18.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T12:12:18.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T12:12:18.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T12:12:18.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T12:12:18.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T12:12:18.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T12:12:18.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2024-09-18T12:12:18.365 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-18T12:12:18.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T12:12:18.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=726a463f-be71-4680-974c-937b599e9da6 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.188934/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T12:12:18.366 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2024-09-18T12:12:18.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T12:12:18.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T12:12:18.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T12:12:18.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T12:12:18.382 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:12:18.381+0000 7f61ebe1b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:12:18.421 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:12:18.419+0000 7f61ebe1b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:12:18.422 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:12:18.420+0000 7f61ebe1b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T12:12:18.423 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:12:18.422+0000 7f61ebe1b440 -1 Falling back to public interface 2024-09-18T12:12:18.431 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-18T12:12:18.430+0000 7f61ebe1b440 -1 osd.0 0 log_to_monitors true 2024-09-18T12:12:18.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T12:12:18.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T12:12:18.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T12:12:18.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T12:12:18.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T12:12:18.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:12:18.593 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-18T12:12:18.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T12:12:18.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:12:18.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:12:18.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T12:12:19.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T12:12:19.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T12:12:19.904 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-18T12:12:19.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T12:12:19.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T12:12:19.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T12:12:20.086 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 26 up_thru 0 down_at 24 last_clean_interval [0,0) [v2:127.0.0.1:6800/206606,v1:127.0.0.1:6801/206606] [v2:127.0.0.1:6802/206606,v1:127.0.0.1:6803/206606] exists,up b4a26b0e-6dfe-4979-8918-031eeecda80b 2024-09-18T12:12:20.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T12:12:20.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T12:12:20.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T12:12:20.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:218: TEST_check_version_health_3: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-18T12:12:20.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-18T12:12:20.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-18T12:12:20.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-18T12:12:20.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-18T12:12:20.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-18T12:12:20.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:12:20.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:12:20.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-18T12:12:20.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-18T12:12:21.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-18T12:12:21.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-18T12:12:21.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-18T12:12:21.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-18T12:12:21.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-18T12:12:21.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-18T12:12:21.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-18T12:12:21.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-18T12:12:21.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-18T12:12:21.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:220: TEST_check_version_health_3: ceph health detail 2024-09-18T12:12:21.784 INFO:tasks.workunit.client.0.smithi157.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph 2024-09-18T12:12:21.784 INFO:tasks.workunit.client.0.smithi157.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2024-09-18T12:12:21.784 INFO:tasks.workunit.client.0.smithi157.stdout: osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test 2024-09-18T12:12:21.784 INFO:tasks.workunit.client.0.smithi157.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2024-09-18T12:12:21.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:221: TEST_check_version_health_3: ceph health detail 2024-09-18T12:12:21.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:221: TEST_check_version_health_3: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-18T12:12:22.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:222: TEST_check_version_health_3: ceph health detail 2024-09-18T12:12:22.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:222: TEST_check_version_health_3: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2024-09-18T12:12:22.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:223: TEST_check_version_health_3: ceph health detail 2024-09-18T12:12:22.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:223: TEST_check_version_health_3: grep -q 'osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test' 2024-09-18T12:12:22.534 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:224: TEST_check_version_health_3: ceph health detail 2024-09-18T12:12:22.535 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:224: TEST_check_version_health_3: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2024-09-18T12:12:22.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2024-09-18T12:12:22.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-18T12:12:22.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T12:12:22.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-18T12:12:22.781 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:12:22.781 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:12:22.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:12:22.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:12:22.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:12:22.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:12:22.888 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:12:22.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:12:22.889 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:12:22.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:12:22.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:12:22.890 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:12:22.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:12:22.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:12:22.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:12:22.892 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:12:22.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:12:22.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T12:12:22.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-18T12:12:22.899 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:12:22.899 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:12:22.900 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:12:22.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188934 2024-09-18T12:12:22.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:12:22.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:12:22.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T12:12:22.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/ver-health 0 2024-09-18T12:12:22.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-18T12:12:22.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T12:12:22.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-18T12:12:22.902 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T12:12:22.902 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T12:12:22.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T12:12:22.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T12:12:22.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T12:12:22.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T12:12:22.903 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T12:12:22.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T12:12:22.904 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T12:12:22.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T12:12:22.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T12:12:22.905 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T12:12:22.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:12:22.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T12:12:22.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T12:12:22.906 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:12:22.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:12:22.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T12:12:22.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-18T12:12:22.909 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T12:12:22.909 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T12:12:22.909 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188934 2024-09-18T12:12:22.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188934 2024-09-18T12:12:22.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T12:12:22.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T12:12:22.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T12:12:22.911 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T12:12:22.912 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T12:12:22.977 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2024-09-18T12:12:22.977 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-18T12:12:23.292 DEBUG:teuthology.parallel:result is None 2024-09-18T12:12:23.292 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T12:12:23.317 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T12:12:23.318 DEBUG:teuthology.orchestra.run.smithi157:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-18T12:12:23.382 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T12:12:23.382 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-18T12:12:23.393 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-18T12:12:23.394 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T12:12:23.470 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-18T12:12:23.478 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-18T12:12:23.479 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T12:12:23.501 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-09-18T12:12:23.505 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T12:12:23.505 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-09-18T12:12:23.505 INFO:teuthology.orchestra.run.smithi157.stdout:^* hv01.front.sepia.ceph.com 3 6 377 44 -756ns[ -18us] +/- 40ms 2024-09-18T12:12:23.505 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 44 -1305us[-1323us] +/- 63ms 2024-09-18T12:12:23.505 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +1328us[+1328us] +/- 63ms 2024-09-18T12:12:23.506 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T12:12:23.506 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-18T12:12:23.515 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-18T12:12:23.516 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-18T12:12:23.523 DEBUG:teuthology.orchestra.run.smithi157:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-18T12:12:23.604 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-18T12:12:23.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T12:12:23.632 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-18T12:12:23.640 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-18T12:12:23.649 INFO:teuthology.task.internal:Duration was 3766.640013 seconds 2024-09-18T12:12:23.649 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-18T12:12:23.657 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-18T12:12:23.657 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T12:12:23.710 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T12:12:24.088 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-18T12:12:24.088 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2024-09-18T12:12:24.088 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T12:12:24.152 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-18T12:12:24.152 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T12:12:24.219 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-18T12:12:24.219 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T12:12:24.358 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-18T12:12:24.367 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-18T12:12:24.368 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T12:12:24.393 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-18T12:12:24.402 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:12:24.456 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2024-09-18T12:12:24.487 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:12:24.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T12:12:24.542 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-18T12:12:24.554 INFO:teuthology.task.internal:Transferring archived files... 2024-09-18T12:12:24.557 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910935/remote/smithi157 2024-09-18T12:12:24.558 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T12:12:24.628 INFO:teuthology.task.internal:Removing archive directory... 2024-09-18T12:12:24.628 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T12:12:24.678 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-18T12:12:24.688 INFO:teuthology.task.internal:Not uploading archives. 2024-09-18T12:12:24.688 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-18T12:12:24.697 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-18T12:12:24.697 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T12:12:24.733 INFO:teuthology.orchestra.run.smithi157.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 12:12 /home/ubuntu/cephtest 2024-09-18T12:12:24.734 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-18T12:12:24.746 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-18T12:12:24.767 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} duration: 3766.6400134563446 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-18T12:12:24.767 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T12:12:24.860 INFO:teuthology.run:pass